Efficient calculation method for big data real-time stream processing

Abstract

The invention relates to the technical field of big data processing, and discloses a high-efficiency calculation method for big data real-time stream processing, which comprises the following steps: generating a predicted workload resource consumption time sequence and a high-frequency state key set for micro-batches, monitoring an available resource time sequence of each calculation node, calculating a comprehensive cost for each candidate node, and calculating a high-frequency state key set for each candidate node; the comprehensive cost weight fuses three components: time sequence matching cost for measuring a resource supply and demand time sequence matching degree, state affinity cost for quantifying data migration overhead and operator deployment cost, a node with the lowest comprehensive cost is selected as an optimal node, calculation is executed at the node after needed state data is migrated, and finally, calculation is performed at the optimal node. And comparing actual and predicted resource consumption of the task, calculating a prediction deviation to perform closed-loop calibration on the prediction model, and continuously improving the decision accuracy. According to the invention, fine matching of resource supply and demand and localization of data processing are realized, the resource utilization rate is improved, and the processing delay is reduced.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, specifically to an efficient computing method for real-time big data stream processing. Background Technology

[0002] Real-time big data stream processing technology, as a key technology for processing continuous, high-speed data streams, has been widely applied in fields such as the Internet of Things, real-time risk control, and online recommendation. In distributed stream processing systems, task scheduling is the core element determining system performance. Its goal is to efficiently allocate computing tasks across the cluster's computing nodes to maximize resource utilization and minimize processing latency.

[0003] However, existing stream processing task scheduling methods still have shortcomings when dealing with complex and ever-changing production environments. First, many scheduling strategies rely primarily on resource snapshot information at the time of scheduling, which ignores the resource consumption patterns of computing tasks over a future period and the dynamic changes in available resources on computing nodes.

[0004] Secondly, for stream processing applications that require state maintenance, existing technologies often fail to tightly couple computation scheduling with the location of state data. Scheduling decisions often do not adequately consider whether the state data accessed by the task is stored locally, resulting in frequent remote state reads and writes over the network during task execution, which severely impacts the real-time response speed of stream processing tasks.

[0005] Furthermore, the characteristics of data flow and the operational state of the cluster are constantly changing dynamically. Traditional scheduling methods often rely on fixed models or heuristic rules, lacking an effective self-learning and adjustment mechanism. When workload patterns drift or cluster states change, the accuracy and effectiveness of scheduling decisions gradually decline over long-term operation, affecting the stability and robustness of the system. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an efficient computing method for real-time big data stream processing. It solves the problems of low resource utilization and high processing latency in distributed stateful stream processing systems, where traditional scheduling methods rely solely on static resource snapshots, ignore state data affinity, and lack the ability to adapt to dynamic changes in workload.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an efficient computing method for real-time big data stream processing, comprising the following steps:

[0008] S1. Aggregate data units in the real-time data stream to form micro-batches, analyze the micro-batches, generate a workload resource profile representing the resource requirements of the micro-batches, and determine the set of associated status keys corresponding to the status data required to process the micro-batches.

[0009] S2. For multiple computing worker nodes in a distributed cluster, predict and generate a node available resource time series that represents the available resources of each computing worker node in the future time window.

[0010] S3. Based on the workload resource profile, the associated status key set, and the node available resource time series, calculate the comprehensive cost value for each of the plurality of computing work nodes, and select the computing work node with the smallest comprehensive cost value as the optimal target computing work node.

[0011] S4. Migrate the state data corresponding to the associated state key set to the optimal target computing node, and execute the computing task for the micro-batch on the optimal target computing node.

[0012] Preferably, in step S1, the generated workload resource profile is a workload resource consumption time series that is generated using a prediction model and characterizes the resource consumption of the micro-batch during processing over time.

[0013] Preferably, in step S4, after the computation task is executed, the actual resource consumption trajectory of the optimal target computation node processing the micro-batch is obtained; the deviation between the workload resource consumption time series and the actual resource consumption trajectory is calculated; and the prediction model is calibrated based on the deviation.

[0014] Preferably, in step S3, calculating the comprehensive cost value for each of the plurality of computing work nodes includes: a time-series matching cost component calculated based on the workload resource consumption time series and the node's available resource time series. The time-series matching cost component is used to quantify the degree of matching between the resource demand curve of the micro-batch and the available resource curve of the computing node over a future period. A specific calculation method is as follows:

[0015]

[0016] Among them, C temporal This represents the time-series matching cost component; the smaller the value, the higher the degree of matching between resource demand and available resources. This indicates a summation operation on all discrete time points within the entire time window; T represents the total number of discrete time points within the predicted time window, which is a positive integer; i is an integer from 1, 2, ..., T; wi R represents the preset weight coefficient at time point i, a positive real number used to adjust the penalty for resource mismatch at different time points; profile (i) represents the predicted resource consumption value of the workload resource consumption time series at time point i; R available (i) represents the predicted available resource value of a node at time point i; (R) profile (i)-R available (i)) 2 This represents the square of the difference between the resource demand value and the available resource value at time point i. This squared term is used to measure the deviation between the two at that time point, and the larger the difference, the more exponentially the value of this term increases.

[0017] Preferably, in step S1, the associated state key set is a high-frequency common access state key set.

[0018] Preferably, in step S3, calculating the comprehensive cost value for each of the plurality of computing work nodes further includes: a state affinity hash cost component calculated based on the location of the state data within the high-frequency common access state key set at the computing work node. This cost component is used to quantify the data movement overhead required to migrate the state data to the target computing node. A specific calculation method is as follows:

[0019]

[0020] Among them, C affinity This represents the state affinity hash cost component. The smaller the value, the higher the affinity between the target computing node and the required state data, and the lower the migration overhead. This represents iterating through and summing the values ​​of each state key in the set of frequently accessed state keys; K hca K represents the set of frequently accessed common state keys; k represents the set K. hca A specific status key; N j L(k,N) represents the j-th computational node currently being evaluated for cost; j ) is a position function used to determine the state data corresponding to state key k relative to the computation worker node N. j Location cost. Specifically, when the state data corresponding to state key k already exists in the computing worker node N. j At that time, L(k,N) j The value of ) is 0; otherwise, its value is a positive number, which is related to the migration of the state data to the computing worker node N. j The required network overhead is proportional.

[0021] Preferably, in step S3, calculating the comprehensive cost value for each of the plurality of computing work nodes further includes: an operator deployment cost component, wherein the operator deployment cost component depends on whether the fusion computing operator is dynamically generated for the high-frequency common access state key set.

[0022] Preferably, the fusion computation operator is a computational unit that combines multiple standard processing steps for the high-frequency common access state key set into a single function call. When a high-frequency common access state key set is identified, this fusion computation operator is dynamically generated and deployed to the target computing node, thereby reducing the number of function calls and the serialization / deserialization overhead of data in intermediate processing stages. The operator deployment cost component quantifies the computational resource overhead required to generate and deploy this operator.

[0023] Preferably, step S4 specifically involves: using a single batch operation for coordinated migration to migrate the state data corresponding to the high-frequency common access state key set to the optimal target computing node, and executing the computation task for the micro-batch on the optimal target computing node. This batch operation merges multiple independent state data migration requests into a single network transmission, reducing the number of handshakes and additional overhead in network communication.

[0024] Preferably, the resources in the workload resource profile and the node available resource time series include at least one of: CPU, memory bandwidth, and network I / O.

[0025] This invention provides an efficient computational method for real-time streaming processing of big data. It has the following beneficial effects:

[0026] 1. This invention enables forward-looking selection of computing nodes by predicting and matching the resource consumption patterns of micro-batch processing with the available resource patterns of computing nodes in a timely manner. This avoids scheduling conflicts and performance bottlenecks caused by delayed resource information and ensures that computing tasks have a stable and matched resource supply throughout the entire execution cycle, thereby improving the overall resource utilization and processing throughput of the distributed cluster.

[0027] 2. This invention identifies a set of frequently accessed state keys and, based on this, performs state data affinity cost calculation, dynamic fusion calculation operator generation, and batch data collaborative migration. This achieves a high degree of collaboration between computing tasks and state data, effectively reducing unnecessary state data network transmission overhead and intermediate function call overhead. It also precisely schedules computing tasks to the optimal data location, reduces end-to-end processing latency, and improves the response speed of real-time stream processing.

[0028] 3. This invention constructs a closed-loop adaptive feedback calibration mechanism by using the deviation between the actual resource consumption trajectory and the predicted consumption time series after the computation task is executed to calibrate the prediction model. This mechanism enables the scheduling method to continuously learn and self-optimize, dynamically adapt to changes in data flow characteristics and cluster status, thereby continuously improving the accuracy of the prediction model and ensuring the efficiency and robustness of scheduling decisions in long-term operation. Attached Figure Description

[0029] Figure 1 This is an overall flowchart of a stream processing task scheduling method according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the comprehensive cost value calculation process according to an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the state data migration process according to an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of a closed-loop feedback and model calibration process according to an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] See attached document Figure 1 , Figure 1 This is an overall flowchart of a stream processing task scheduling method according to an embodiment of the present invention. The present invention provides an efficient computing method for real-time big data stream processing, executed in a distributed computing cluster environment consisting of multiple computing worker nodes, including:

[0035] Step S1: Analyze the real-time aggregated data micro-batch to generate workload resource profiles that represent future resource needs, and determine the set of associated state keys corresponding to the state data that needs to be accessed during the computation process.

[0036] Step S2: Predict the future available resources for each computing worker node in the cluster and generate a time series of available node resources.

[0037] Step S3: Based on the previously generated workload resource profile, associated state key set, and node available resource time series, calculate a comprehensive cost value for each candidate computing node. This comprehensive cost value integrates costs from multiple dimensions, including resource time series matching degree and state data migration overhead. Subsequently, select the computing node with the minimum comprehensive cost value as the optimal target computing node for executing this computing task.

[0038] Step S4: Migrate the state data corresponding to the determined set of associated state keys to the optimal target computing node, and execute the micro-batch computing task on the optimal target computing node.

[0039] In a preferred embodiment, after the task is completed, actual resource consumption data is collected for comparison with the predicted data, and the prediction model is calibrated based on the deviation, thereby forming a closed-loop adaptive optimization process.

[0040] See attached document Figure 1 The S1 step of the method of the present invention will be described in detail below. The method of the present invention receives a continuous, unbounded stream of data units from one or more data sources. These data units are temporarily stored in a buffer. Using a preset aggregation strategy, data units are extracted from the buffer to form structured micro-batches, i.e., the aggregation process of data units.

[0041] In one embodiment, a time-window-based aggregation strategy is employed. A timer is started, with its period set to a preset time interval Δt. When the timer expires, all data units received in the buffer since the previous time point are aggregated to form a new micro-batch. Subsequently, the data units corresponding to the newly formed micro-batch are removed from the buffer, the timer is reset, and the next aggregation cycle begins. The value of Δt is a configurable parameter, such as 100 milliseconds.

[0042] In another embodiment, a data volume-based aggregation strategy is employed, maintaining a counter to record the number of data units entering the buffer since the last aggregation. The counter counts until it reaches a preset threshold N. threshold At that time, the earliest N to enter the buffer threshold Each data unit is aggregated to form a new micro-batch. Then, the counter is subtracted by N. threshold Alternatively, it can be reset to zero after the data is removed. threshold The value is a configurable parameter, such as 1000 data units.

[0043] In yet another embodiment, a sliding window strategy is employed. The sliding window strategy defines two parameters: window length W. len and sliding step size S stepBoth are units of time. A micro-batch is generated from the past W... len It consists of data units arriving within a time period, and every S step A new micro-batch is generated at each time interval.

[0044] Regardless of the aggregation strategy employed, the output of this data unit's aggregation process step is a structured data set, known as a micro-batch. This micro-batch is then passed to subsequent analysis stages in step S1 to generate workload resource profiles and determine the set of associated state keys.

[0045] After forming micro-batches according to the aforementioned aggregation strategy, step S1 then analyzes the micro-batches, one of the purposes of which is to generate workload resource profiles that characterize the resource requirements of the micro-batches in future processing.

[0046] In a preferred embodiment, the workload resource profile is specifically represented as a workload resource consumption time series. This workload resource consumption time series is not based on instantaneous measurements of the current state, but rather is generated using a predictive model to forecast resource consumption over a future period.

[0047] The input to the prediction model is a feature vector F extracted from the current micro-batch. batch . Eigenvector F batch This includes: the total number of data units contained within a micro-batch, the distribution statistics of each data type, the average size of the data units, and other quantitative indicators describing the inherent complexity of the data. In addition, the input may include historical performance data related to similar micro-batch processing in the past, for model reference.

[0048] In one specific implementation, the prediction model is a pre-trained recurrent neural network, such as a long short-term memory network or a gated recurrent unit network. The prediction model learns the mapping relationship between micro-batch features and their future resource consumption patterns by training on a large amount of historical (feature vectors, actual resource consumption trajectories) data pairs.

[0049] The output of the prediction model is the workload resource consumption time series, which consists of T consecutive predicted resource consumption values, and can be represented as follows:

[0050]

[0051] Where T represents the total number of discrete time points within the predicted time window, which is a positive integer; R profile (i) represents the predicted resource consumption value of the workload resource consumption time series at the i-th time point; i is an integer in 1, 2, ..., T.

[0052] The resources corresponding to the resource consumption value include at least one of CPU utilization, memory bandwidth usage, and network I / O throughput.

[0053] The generated workload resource consumption time series will be used in the subsequent S3 step as the calculation time series matching cost component C. temporal One of the key inputs.

[0054] In step S1, parallel to generating the workload resource profile, the set of associated state keys corresponding to the state data required to process the micro-batch is determined. This process first statically or dynamically analyzes the computational logic to be triggered for each data unit constituting the micro-batch. For each data unit, its processing flow is parsed, identifying all entries in the distributed state store that need to be read or written. Each such entry is identified by a unique state key. All unique state keys required to process all data units within the micro-batch are collected to form an initial set, which is the set of associated state keys.

[0055] In a preferred embodiment, the aforementioned set of associated state keys is further analyzed to identify the set of frequently accessed state keys. This identification process is based on mining historical state access patterns. A historical access database is maintained, which records the set of state keys accessed when processing each micro-batch in the past.

[0056] Specifically, an association rule mining algorithm is applied. This algorithm works by setting a minimum support threshold (supp). min The state key combinations that appear more frequently than the minimum support threshold are identified from historical records; these combinations are called frequent itemsets.

[0057] After identifying all frequent itemsets, the initial set of associated state keys for the current micro-batch is matched against the frequent itemsets. The frequent itemset with the highest matching degree is determined as the set of high-frequency common access state keys required for this calculation, denoted as K. hca .

[0058] This step generates a set of associated state keys (or its preferred form, a set of high-frequency co-access state keys K). hca This is one of the key inputs for calculating the overall cost value in the subsequent S3 step. Specifically, it is used to calculate the state affinity hash cost component C. affinity The state affinity hash cost component quantifies the cost of migrating the required state data to a candidate computing node. The state affinity hash cost component C... affinity The calculation method is as follows:

[0059]

[0060] Among them, C affinity This represents the state affinity hash cost component. The smaller the value, the higher the affinity between the target computing node and the required state data, and the lower the migration overhead. This represents iterating through and summing the values ​​of each state key in the set of frequently accessed state keys; K hca K represents the set of frequently accessed common state keys; k represents the set K. hca A specific status key; N j L(k,N) represents the j-th computational node currently being evaluated for cost; j ) is a position function used to determine the state data corresponding to state key k relative to the computation worker node N. j Location cost. Specifically, when the state data corresponding to state key k already exists in the computing worker node N. j At that time, L(k,N) j The value of ) is 0; otherwise, its value is a positive number, which is related to the migration state data to the computing worker node N. j The required network overhead is proportional.

[0061] See attached document Figure 1 Step S2 of this invention aims to generate a time series of available resources for each computing worker node in the distributed cluster. The initial step involves collecting status data from each computing worker node in real time and periodically.

[0062] In one specific embodiment, the distributed cluster comprises multiple computing worker nodes. In each computing worker node N... j On each of these systems, a lightweight monitoring agent is deployed and runs. The monitoring agent's function is to perform a sampling at a preset period Δt. sample (For example, 1 second), read a set of raw performance metrics from the operating kernel or hardware performance monitor of the node where it resides.

[0063] For CPU resources, the monitoring agent does not simply collect a general utilization value, but instead queries the CPU time statistics interface provided by the operation to obtain the precise time slice counts consumed by the CPU in different states (e.g., user mode, system mode, idle mode, I / O wait mode). This is achieved by comparing the time slice count at time point t with t-Δt. sample The difference between the two readings can be used to calculate the percentage of time the CPU spent on non-idle tasks during the previous sampling period, thus obtaining an accurate CPU load value.

[0064] For memory resources, in addition to collecting the usual available physical memory size, this embodiment also collects memory bandwidth utilization. The monitoring agent directly measures the memory bandwidth utilization during the sampling period Δt by accessing hardware performance counters. sampleWithin this range, the total amount of data read and written through the memory controller. Divide this data amount by the preset sampling period Δt. sample This gives the actual memory bandwidth usage in GB / s, directly reflecting the level of contention on the memory bus.

[0065] For network I / O resources, the monitoring agent queries the statistics maintained by the kernel for each network interface, recording the cumulative number of bytes received and the cumulative number of bytes sent. Similarly, it calculates the data at time points t and t-Δt. sample The difference between the two readings is divided by the preset sampling period Δt. sample This allows us to obtain the average network receive throughput and transmit throughput within that period.

[0066] At the end of each sampling period, the monitoring agent encapsulates the collected and calculated set of raw performance metrics (e.g., CPU load, available physical memory, memory bandwidth usage, network receive throughput, network send throughput) into a timestamped raw metric vector. This raw metric vector is then sent to a central prediction component as raw input data to generate time series data of available resources for the nodes. This collection process is performed synchronously and continuously on all computing worker nodes.

[0067] The central prediction component receives data from each computing worker node N. j After the raw indicator vectors are periodically sent, these raw indicators are first preprocessed and converted into quantified values ​​representing available resources. This conversion is based on the hardware specifications of each computing worker node or its theoretical maximum capacity. For memory bandwidth, the available memory bandwidth is calculated as the maximum theoretical memory bandwidth B of that computing worker node. max Subtract the collected memory bandwidth usage value. After this step, for each computing node N j Each of these generates a historical, time-series vector data stream of available resources.

[0068] Next, for each computational working node N j It uses an independent predictive model specific to the computing node to process the historical available resource vector data stream and generate a node available resource time series that characterizes the available resources within a future time window.

[0069] In one specific implementation, the predictive model is an autoregressive integral moving average model or a pre-trained recurrent neural network. The predictive model predicts future trends in available resources by learning historical patterns of available resource changes in individual nodes (e.g., resource fluctuations due to periodic background tasks).

[0070] The input to the prediction model is the computation of worker node N.j The model outputs a time series of available resource vectors over a past period (e.g., the past few hundred sampling periods). This time series consists of T consecutive predicted available resource values ​​and can be represented as follows:

[0071]

[0072] Where T is the total number of discrete time points within the prediction time window, a positive integer, and consistent with the length of the workload resource consumption time series generated in step S1; R available (i) is the computation worker node N j The predicted available resource value at time point i; i is an integer from 1, 2, ..., T.

[0073] The types of available resource values ​​predicted (e.g., CPU, memory bandwidth) also correspond to the types of resources predicted in the workload resource consumption time series.

[0074] This step generates a time series of available resources for each candidate computing worker node, which will be used as the computation time series matching cost component C in the subsequent S3 step. temporal The core input.

[0075] See attached document Figure 1 After generating the workload resource profile and associated state key set in step S1, and generating the node available resource time series for each computing worker node in step S2, the method of this invention proceeds to step S3. The core task of step S3 is to calculate a comprehensive cost value for each candidate computing worker node in the distributed cluster, and select the optimal target computing worker node based on the comprehensive cost value.

[0076] For each candidate compute worker node N in the cluster j Each of these will be calculated as an independent comprehensive cost value C. total (N j The overall cost value is not evaluated based on a single dimension, but rather through a multi-component cost model that quantifies and combines the suitability of multiple different aspects of the computational work nodes.

[0077] See attached document Figure 2 , Figure 2 This is a schematic diagram illustrating the comprehensive cost value calculation process according to an embodiment of the present invention. In a specific embodiment, the comprehensive cost value is composed of three independent cost components through a weighted summation: a time-matched cost component C. temporal A state affinity hash cost component C affinity And an operator deployment cost component C operatorComprehensive cost value C total (N j The specific calculation is as follows:

[0078] C total (N j )=α·C temporal +β·C affinity +γ·C operator ;

[0079] Among them, C total (N j 0 indicates that the computation is performed on worker node N. j The smaller the comprehensive cost value, the more accurate it is to calculate the working node N. j The more suitable it is for executing the current micro-batch computation task; C temporal It is for computing worker node N j Calculated time-matching cost component; C affinity It is for computing worker node N j Calculated state affinity hash cost component; C operator It is for computing worker node N j The evaluated operator deployment cost components; α, β, γ are preset weighting coefficients used to adjust the relative importance of each cost component. These coefficients are all positive real numbers, and in one embodiment, their sum is normalized to 1 (i.e., α + β + γ = 1). By adjusting the values ​​of these coefficients, the scheduling strategy can be made to focus on different objectives such as resource timing matching, data locality, or computational optimization.

[0080] After calculating the comprehensive cost value for all candidate computing nodes, the final step of step S3 is to compare all these cost values ​​and select the computing node with the minimum cost value as the optimal target computing node.

[0081] In step S3, the working node N is calculated. j The comprehensive cost value C total (N j The first step is to determine the timing matching cost component C. temporal The purpose of calculating the time-series matching cost component is to quantify the predicted resource consumption and computational work nodes N of the micro-batch to be processed within a future time window consisting of T discrete time points. j The degree of mismatch between predicted available resources.

[0082] The input to this calculation process is the two time series generated in the preceding steps: the workload resource consumption time series associated with the micro-batch, generated in step S1. And the computation working node N generated in step S2 j Associated node available resource time series

[0083] Timing matching cost component C temporal The specific calculation is achieved by taking the weighted sum of squares of the differences between the two time series at each corresponding time point. The calculation formula is as follows:

[0084]

[0085] Among them, C temporal This represents the time-series matching cost component; the smaller the value, the higher the degree of matching between resource demand and available resources. This indicates a summation operation on all discrete time points within the entire time window; T represents the total number of discrete time points within the predicted time window, which is a positive integer; i is an integer from 1, 2, ..., T; w i R represents the preset weight coefficient at time point i, a positive real number used to adjust the penalty for resource mismatch at different time points; profile (i) represents the predicted resource consumption value of the workload resource consumption time series at time point i; R available (i) represents the predicted available resource value of a node at time point i; (R) profile (i)-R available (i)) 2 This represents the square of the difference between the resource demand value and the available resource value at time point i. This squared term is used to measure the deviation between the two at that time point, and the larger the difference, the more exponentially the value of this term increases.

[0086] In step S3, the time-matching cost component is calculated immediately following the process, and the method will be used for candidate computation worker nodes N. j The second component for calculating the overall cost value: the state affinity hash cost component C. affinity The purpose of calculating the state affinity hash cost component is to quantify and concentrate all the state data required to process the current micro-batch onto the computation worker node N. j The resulting data migration overhead.

[0087] The core input to this calculation process is the set of high-frequency common access state keys K identified in step S1. hca To support this computation, a metadata directory is maintained in the distributed cluster. This metadata directory records in real time and accurately the physical computing worker node where the state data corresponding to each state key k is currently located.

[0088] For candidate computing nodes N j The calculation of the cost component of state affinity hashing is achieved by traversing the set K of frequently accessed state keys. hcaFor each state key k, query the metadata directory to determine its location and accumulate its value relative to the compute worker node N. j This is achieved through location costs.

[0089] The formula for calculating the cost component of state affinity hashing is as follows:

[0090]

[0091] Among them, C affinity This represents the state affinity hash cost component. The smaller the value, the higher the affinity between the target computing node and the required state data, and the lower the migration overhead. This represents iterating through and summing the values ​​of each state key in the set of frequently accessed state keys; K hca K represents the set of frequently accessed common state keys; k represents the set K. hca A specific status key; N j L(k,N) represents the j-th computational node currently being evaluated for cost; j ) is a position function used to determine the state data corresponding to state key k relative to the computation worker node N. j Location cost. Specifically, when the state data corresponding to state key k already exists in the computing worker node N. j At that time, L(k,N) j The value of ) is 0; otherwise, its value is a positive number, which is related to the migration state data to the computing worker node N. j The required network overhead is proportional.

[0092] In step S3, to calculate the working node N j The third component in calculating the overall cost value is the operator deployment cost component C. operator The determination of the operator deployment cost components aims to quantify the impact on computational worker nodes N. j The above represents the instantaneous resource overhead incurred in preparing the execution environment for the current micro-batch computation task.

[0093] In one specific implementation, for each micro-batch to be processed, the method dynamically generates one or more fusion computation operators based on its data characteristics and processing logic. A fusion computation operator is a piece of optimized, executable code that combines multiple data processing steps into a single execution unit. The computation task is then scheduled to the target computation worker node N. j Previously, it was necessary to ensure that the fusion computing operator was available on the node. This process was called operator deployment, which itself consumed CPU and memory resources.

[0094] Operator deployment cost component C operatorThe determination of is not achieved through a complex continuous function calculation, but rather through a cost evaluation function based on state queries. The cost evaluation function assesses the deployment of the fusion computation operators required for the current micro-batch to computation worker nodes N. j The difficulty.

[0095] Specifically, each computing worker node N j Each maintains a local cache, which stores instances of recently used fusion computation operators or their compiled binary code. When N is a computation worker node... j When evaluating the deployment cost of operators, the method examines the computation worker nodes N. j Does the local cache contain an operator instance that is exactly the same as or binary compatible with the operator required for the current micro-batch?

[0096] Operator deployment cost component C operator The rules for determining it are as follows:

[0097] C operator =F deploy (O batch N j );

[0098] Among them, C operator This indicates that the computation is performed on worker node N. j The operator deployment cost component is a non-negative real number; O batch N represents the specific fusion computation operator generated for processing the current micro-batch; j F represents the j-th computational node currently having its cost evaluated; deploy (O batch N j ) is a deployment cost evaluation function, and its logic is as follows:

[0099] If we are calculating the working node N j Operator O was found in the local cache. batch If a usable instance (i.e., a cache hit) is found, the cost evaluation function returns a lower preset constant value c. warm This represents the small overhead of loading operators from the cache and initializing their execution context.

[0100] If we are calculating the working node N j Operator O was not found in the local cache. batch If no instance is available (i.e., a cache miss), the cost evaluation function returns a higher preset constant value c. cold This indicates that operator code needs to be transmitted over a network during computation at worker node N. j This involves just-in-time compilation and the full overhead of allocating entirely new memory and execution resources to it. In one embodiment, ccold The value is much greater than c warm The value of .

[0101] After determining C operator After the value of C mentioned above, temporal and C affinity One step is used to calculate the final comprehensive cost value C. total (N j ).

[0102] In the final step of S3, after the comprehensive cost value of all candidate computing worker nodes in the distributed cluster has been calculated according to the aforementioned method, the method of the present invention will perform the selection operation of the optimal target computing worker node.

[0103] The selection operation is a deterministic comparison process. The method iterates through all calculated comprehensive cost values ​​and finds the entry with the minimum value. The computation node corresponding to the minimum comprehensive cost value is determined as the optimal target computation node for executing the current micro-batch computation task.

[0104] This selection process can be precisely described by the following expression:

[0105]

[0106] Where, N optimal This represents the optimal target computation working node that has been selected, which is the final output of step S3. It is a minimization operator whose function is to return a value that minimizes the objective function C. total The parameter that yields the minimum value is the calculation of the working node N. j θ represents the set of all candidate computing worker nodes participating in this selection process in the distributed cluster; N j C represents the j-th computational node currently being evaluated for cost; total (N j ) indicates that the computation is performed on worker node N. j The smaller the comprehensive cost value, the more accurate it is to calculate the working node N. j It is more suitable for performing the current micro-batch computation task.

[0107] Optimal target calculation working node N optimal The identifier is then passed to step S4 as the target location for state data migration and computation tasks.

[0108] See attached document Figure 3 , Figure 3 This is a schematic diagram of the state data migration process according to an embodiment of the present invention. In step S3, the optimal target computation node N is determined. optimalNext, proceed to step S4. The primary task is to perform state data migration to ensure that the data required for performing computational tasks is concentrated on the optimal target computational worker node, thereby achieving localized data processing.

[0109] The direct input to this migration process is the output N of step S3. optimal and the set K of high-frequency common access state keys determined in step S1. hca In addition, the process relies on a metadata directory that can be queried and updated in real time. The metadata directory records the current physical node location of each status key and its corresponding status data.

[0110] For the set of high-frequency common access state keys K hca For each state key k in the table, the method first queries the metadata directory to determine its current source compute worker node N. source .

[0111] Subsequently, the source computing worker node N source The identifier and the optimal target calculation working node N optimal The identifiers are compared. If they do not match, i.e., N source ≠N optimal This triggers a data migration operation. This operation transfers the complete state data block associated with state key k from the source compute worker node N via the network. source The data is directly transferred from the storage to the optimal target computing node N. optimal In the local state storage.

[0112] After the data migration is complete and confirmed, the method will atomically update the metadata catalog, changing the entry for status key k to N. optimal If the source computing worker node N source The identifier and the optimal target calculation working node N optimal If the identifiers match, it indicates that the required data is already local and no migration operation is required.

[0113] This physical data migration operation is the state affinity hash cost component C in step S3. affinity The calculation directly corresponds to this. In the cost calculation stage, the location function L(k,N) j The case where it returns a non-zero value indicates a scenario where network data transmission needs to be performed.

[0114] The entire state data migration process is for set K hca All non-local state keys are executed in parallel to minimize preparation time. The method waits until all necessary data migration operations are completed before proceeding to the next stage, step S4, which involves computation on the optimal target node N. optimalIt performs the actual computational tasks.

[0115] In step S4, after all necessary state data migration operations are completed, the method then proceeds to the determined optimal target computation node N. optimal On top, start and execute the computation task for the micro-batch.

[0116] The execution of this computational task specifically refers to the computation of the optimal target work node N. optimal In the execution environment, the fusion computation operator O generated for the current micro-batch is invoked. batch The execution context of the fusion computation operator has been prepared, and the overhead of this preparation process has been borne by the operator deployment cost component C in step S3. operator Quantification was performed.

[0117] Fusion computation operator O batch It uses all data units contained within a micro-batch as its primary input. During execution, the fusion computation operator O... batch These data units can be processed sequentially or in parallel. For each data unit, the processing logic needs to read or update one or more state entries.

[0118] Due to the state data migration in the previous stage, the set of high-frequency common access state keys K required to process the current micro-batch is... hca All corresponding state data have been guaranteed to exist in the optimal target computation working node N. optimal It is stored in the local state storage. Therefore, when the fusion computation operator O... batch When performing any state access operation, the operation is performed as a read or write operation to local memory or local storage, thus completely avoiding the latency caused by remote data access across the network.

[0119] Fusion computation operator O batch After execution, two types of outputs are generated: one is the updated state, which is directly and atomically applied to the optimal target computation worker node N. optimal The data is stored in the local state storage; the second is the downstream data generated from the processing logic, which will be sent to subsequent processing stages or external systems.

[0120] When the fusion computation operator O batch After processing all data units within the micro-batch, step S4 is complete.

[0121] See attached document Figure 4 , Figure 4 This is a schematic diagram of the closed-loop feedback and model calibration process according to an embodiment of the present invention. In step S4, the working node N is calculated at the optimal target. optimalAfter executing and completing the micro-batch computation task, the method of the present invention may further include acquiring the actual resource consumption trajectory, comparing it with the prediction, and calibrating the prediction model to form a closed-loop optimization process. The starting point of the closed-loop optimization process is to synchronously acquire the actual resource consumption trajectory during the execution of the computation task.

[0122] Specifically, when step S4 is performed at the optimal target computation working node N... optimal Start the fusion computing operator O batch When performing computational tasks, a lightweight monitoring agent deployed on the node is also activated to perform fine-grained monitoring of resource consumption during the execution of specific operators.

[0123] The lightweight monitoring agent uses the same sampling period Δt as in step S2. sample It runs, but its monitoring scope is not the entire computing node. Instead, it precisely locks onto the fusion computing operator O through the interface provided by the operation. batch The monitoring agent collects raw performance metrics for a specific process or process group during each sampling period, including CPU time consumed by the process, memory bandwidth usage, and network I / O traffic generated.

[0124] This monitoring process starts from the fusion computing operator O batch Execution begins at the start time and continues until its complete termination. Throughout the execution, a series of raw performance metric measurements collected by the monitoring agent and ordered by time are aggregated and processed into a time series consistent with the workload resource consumption time series format generated in step S1. This time series represents the actual resource consumption trajectory.

[0125] For the optimal objective calculation of working node N optimal The actual resource consumption trajectory obtained from the completed computation task. It can be represented as:

[0126]

[0127] Among them, R actual (i) represents the resource consumption value actually observed and measured at the i-th time point during the execution of the computation task. The resource type (e.g., CPU, memory bandwidth) of this value corresponds to the resource type predicted in step S1. T′ represents the total number of discrete time points corresponding to the actual execution time of the computation task, which is a positive integer.

[0128] This step generates the actual resource consumption trajectory. Compared with the workload resource consumption time series predicted for this micro-batch in step S1 Together, they constitute the core input for the next step, model calibration.

[0129] After obtaining the actual resource consumption trajectory of the computation task, the method compares the actual trajectory with the predicted workload resource consumption time series generated for the micro-batch in step S1 to calculate a quantified prediction bias.

[0130] This deviation calculation process takes two time series as input: the predicted workload resource consumption time series. and the actual resource consumption trajectory

[0131] By calculating the sum of the squares of the differences between two time series at corresponding time points, a total prediction error value E can be obtained. The formula for calculating the prediction error value is as follows:

[0132]

[0133] Where E represents the total error of this prediction, which is a non-negative real number. The smaller the value, the higher the accuracy of the prediction; This represents a summation operation whose upper bound is determined by min(T,T′), which takes the smaller of the total number of predicted time points T and the total number of actual execution time points T′, to ensure that comparisons are made only within a common timeframe defined for both sequences; R profile (i) represents the predicted resource consumption value of the workload resource consumption time series at time point i; R actual (i) is the actual measured resource consumption value at time point i, obtained in the previous stage, representing the actual resource consumption trajectory; (R) profile (i)-R actual (i)) 2 This represents the square of the difference between the predicted resource consumption value and the actual measured resource consumption value at time point i.

[0134] The calculated prediction error value E is then used as a loss signal to calibrate and optimize the prediction model used to generate workload resource profiles. The prediction model (e.g., a deep neural network) contains a set of adjustable internal parameters.

[0135] The calibration process specifically involves providing the input data of the observation (i.e., the features of the micro-batch) and the calculated prediction error value E to an optimization algorithm (e.g., stochastic gradient descent or a variant thereof). This algorithm makes a small adjustment to the internal parameters of the prediction model based on the magnitude and direction of the prediction error value E. The goal of this adjustment is to ensure that when the model encounters a micro-batch with similar features in the next observation, the predicted resource consumption time series it generates will more closely approximate the actual consumption trajectory, thereby reducing the calculated error value E.

[0136] This calibration step is repeated after each or every few task executions, thereby continuously and adaptively improving the accuracy of resource consumption prediction in step S1, thus constituting the closed-loop optimization process of this invention.

Claims

1. A highly efficient computational method for real-time big data stream processing, characterized in that, Includes the following steps: S1. Aggregate data units in the real-time data stream to form micro-batches, analyze the micro-batches, generate a workload resource profile representing the resource requirements of the micro-batches, and determine the set of associated status keys corresponding to the status data required to process the micro-batches. S2. For multiple computing worker nodes in a distributed cluster, predict and generate a node available resource time series that represents the available resources of each computing worker node in the future time window. S3. Based on the workload resource profile, the associated status key set, and the node available resource time series, calculate the comprehensive cost value for each of the plurality of computing work nodes, and select the computing work node with the smallest comprehensive cost value as the optimal target computing work node. S4. Migrate the state data corresponding to the associated state key set to the optimal target computing node, and execute the computing task for the micro-batch on the optimal target computing node.

2. The efficient computing method for real-time big data stream processing according to claim 1, characterized in that, In step S1, the generated workload resource profile is a time series of workload resource consumption that is predicted using a prediction model and represents the change of resource consumption over time during the processing of the micro-batch.

3. The efficient computing method for real-time big data stream processing according to claim 2, characterized in that, In step S4, after the computation task is executed, the actual resource consumption trajectory of the optimal target computation work node in processing the micro-batch is obtained. Calculate the deviation between the workload resource consumption time series and the actual resource consumption trajectory; The prediction model is then calibrated based on the deviation.

4. The efficient computing method for real-time big data stream processing according to claim 2, characterized in that, In step S3, calculating the comprehensive cost value for each of the plurality of computing work nodes includes: A time-series matching cost component calculated based on the time series of the workload resource consumption and the time series of the available resources of the node.

5. The efficient computing method for real-time big data stream processing according to claim 1, characterized in that, In step S1, the associated state key set is a high-frequency common access state key set.

6. The efficient computing method for real-time big data stream processing according to claim 5, characterized in that, Step S3, calculating the comprehensive cost value for each of the plurality of computing work nodes, further includes: A state affinity hash cost component is calculated based on the state data within the set of high-frequency common access state keys at the location of the computing work node.

7. The efficient computing method for real-time big data stream processing according to claim 5, characterized in that, Step S3, calculating the comprehensive cost value for each of the plurality of computing nodes, further includes: An operator deployment cost component, wherein the operator deployment cost component depends on whether the fusion computing operator is dynamically generated for the high-frequency common access state key set.

8. The efficient computing method for real-time big data stream processing according to claim 7, characterized in that, The fusion computation operator is a computational unit that combines multiple standard processing steps for the high-frequency common access state key set into a single function call.

9. The efficient computing method for real-time big data stream processing according to claim 5, characterized in that, The S4 step is specifically as follows: Using a single batch operation for collaborative migration, the state data corresponding to the high-frequency common access state key set is migrated to the optimal target computing node, and the computing task for the micro-batch is executed on the optimal target computing node.

10. The efficient computing method for real-time big data stream processing according to claim 1, characterized in that, The workload resource profile and the resources in the node available resource time series include at least one of: CPU, memory bandwidth, and network I / O.

Images