Streaming data query optimization method based on incremental calculation and dynamic load balancing

Abstract

The invention relates to the technical field of stream data processing, in particular to a stream data query optimization method based on incremental calculation and dynamic load balancing, and the control method comprises the steps: S1, constructing a multi-dimensional stream data feature model, and determining the weight of each feature dimension; s2, collecting initial stream data, calculating an incremental calculation granularity coefficient, and determining an initial incremental window; s3, executing an incremental calculation algorithm: collecting stream data in real time, and updating characteristic values; s4, executing a dynamic load balancing algorithm; s5, receiving a user query request; and S6, monitoring streaming data characteristics and node loads in real time, and circularly executing the steps S3-S5 to optimize the query efficiency. The streaming data query optimization method based on incremental calculation and dynamic load balancing is high in stability and reliability, system instability caused by load adjustment lag or migration failure is avoided through mechanisms such as load prediction and early warning and migration verification, and continuity and reliability of streaming data processing are ensured.

Description

Technical Field

[0001] This application relates to the field of streaming data processing technology, and in particular to a streaming data query optimization method based on incremental calculus and dynamic load balancing. Background Technology

[0002] In the era of big data, streaming data is widely present in fields such as financial transactions, IoT sensing, and real-time monitoring. It is characterized by its real-time nature, continuity, massive volume, and dynamic changes. Streaming data queries require real-time responses to meet the business's need for immediate decision-making.

[0003] In existing streaming data query optimization techniques, incremental calculation methods are mostly based on simple difference calculations or fixed window update mechanisms, which have the following drawbacks: First, they do not fully consider the multi-dimensional characteristics of streaming data (such as data generation frequency, data value fluctuation range, and query relevance), resulting in unreasonable incremental calculation granularity, excessive repetitive calculations in some scenarios, and increased query latency; Second, the incremental update triggering conditions are singular, based only on time thresholds or data volume thresholds, and cannot be adaptively adjusted according to dynamic changes in data, resulting in resource waste or decreased query accuracy.

[0004] Meanwhile, existing dynamic load balancing technologies mostly adopt static hash sharding and round-robin scheduling, which have problems such as lagging load perception and high load migration costs. On the one hand, scheduling based solely on historical load data cannot predict the short-term trend of streaming data load changes, resulting in load balancing adjustments lagging behind actual load changes, with some nodes overloaded while others are idle. On the other hand, the load migration process does not consider data correlation, which can easily lead to the data required for queries being scattered across multiple nodes, increasing data transmission overhead and further reducing query efficiency.

[0005] To address the aforementioned issues, we propose an incremental computation and dynamic load balancing-based streaming data query optimization method based on existing technologies. Summary of the Invention

[0006] This application provides a streaming data query optimization method based on incremental calculus and dynamic load balancing to improve the following technical issues.

[0007] This application provides a streaming data query optimization method based on incremental calculus and dynamic load balancing. The control method includes:

[0008] S1. Construct a multi-dimensional streaming data feature model and determine the weights of each feature dimension;

[0009] S2. Collect initial stream data, calculate the incremental calculation granularity coefficient, and determine the initial incremental window;

[0010] S3. Execute the incremental calculation algorithm: collect streaming data in real time, update feature values, trigger incremental calculation through the "feature threshold + prediction trigger" mechanism, adjust the incremental window and complete the incremental calculation;

[0011] S4. Execute dynamic load balancing algorithm: Calculate node load values ​​based on incremental calculation results, predict load through XGBoost model, and perform load migration for nodes that exceed the warning threshold according to the "association priority - minimum cost" strategy.

[0012] S5. Receive user query requests, prioritize retrieving data from incremental calculation results, and if historical data is required, associate it with the stored complete data and return the query results;

[0013] S6. Monitor streaming data characteristics and node load in real time, and repeatedly execute S3-S5 to optimize query efficiency.

[0014] Furthermore, in S1, each feature dimension is defined based on the dynamic characteristics of the streaming data. Each feature dimension is as follows:

[0015] Data generation frequency : This represents the number of data entries generated per unit of time, calculated using the following formula: ,in, For time window Number of data stream entries within;

[0016] Data value fluctuation range Time window The maximum deviation rate of the internal data values ​​is calculated using the following formula: ,in, For the first in the window The value of each data item This represents the average value of the data within the window.

[0017] Query relevance The percentage of times this data stream is associated with historical query statements is calculated using the following formula: ,in, This represents the number of times the data stream has been associated with query statements in the past hour. This represents the total number of all query statements in the past hour.

[0018] Data priority P: The weight value is set according to business requirements, and the value range is [1,5], where 5 represents the highest priority and 1 represents the lowest priority. The priority decreases one by one from 5 to 1.

[0019] Furthermore, in S1, the Analytic Hierarchy Process (AHP) is used to determine the weights of each feature dimension, specifically as follows:

[0020] A1. Constructing the judgment matrix: Five experts in the field of data processing were invited to compare the importance of the four feature dimensions pairwise. A judgment matrix was constructed using the 1-9 scale method, where 1 represents equal importance and 9 represents extreme importance, with importance increasing sequentially from 1 to 9.

[0021] A2. Consistency Test: Calculate the largest eigenvalue of the judgment matrix. And calculate the consistency index. ,in, Let be the number of feature dimensions, if If so, then the matrix satisfies the consistency requirement;

[0022] A3. Weight Determination: Normalize the eigenvectors of the judgment matrix to obtain the weights for each feature dimension. ,in, Data generation frequency Data value fluctuation range Query relevance And the weights normalized to the data priority P.

[0023] Furthermore, in step S2, an incremental calculation granularity coefficient is defined based on the calculation results of the multi-dimensional streaming data feature model. The calculation formula is: Among them, the incremental calculation granularity coefficient The range of values ​​is This corresponds to 5 incremental calculation granularity levels, and the 5 incremental calculation granularity levels are as follows:

[0024] Level 1 Coarse-grained incremental calculation uses a 10-minute time window and only updates the statistical results of the data within the window.

[0025] Level 2 : Coarse-grained incremental calculation, using a 5-minute time window, incrementally updating the statistical results and key data items within the window;

[0026] Level 3 Medium-granularity incremental calculation uses a 2-minute time window to incrementally update all data items within the window;

[0027] Level 4 : Fine-grained incremental calculation, using a 1-minute time window, incrementally updating all data items and data relationships within the window;

[0028] Level 4 Fine-grained incremental calculation uses a 30-second time window to update the data in real time within the window, while retaining historical version data.

[0029] Furthermore, the strategy of the "feature threshold + prediction trigger" mechanism in S3 is as follows:

[0030] Feature threshold triggering strategy: Incremental update is triggered when the change in a certain feature dimension exceeds a preset threshold;

[0031] Prediction triggering strategy: Construct a streaming data feature prediction model based on Long Short-Term Memory (LSTM) networks, and input feature data from the past 10 time windows. It predicts the feature values ​​for the next window. If the prediction results show that a certain feature will exceed the threshold, it triggers incremental calculation in advance.

[0032] Furthermore, the specific strategy of the incremental calculation algorithm in S3 is as follows:

[0033] B1. Initialization: Obtaining the initial characteristic values ​​of the streaming data Calculate the initial particle size coefficient Determine the initial incremental calculation window ;

[0034] B2. Data Acquisition, within the Time Window Internally collected streaming data, recording the number of data entries. Data values and the number of related queries ;

[0035] B3. Feature update: Calculate the feature values ​​for the current window. ,like If there are no business adjustments, the status quo will remain unchanged.

[0036] B4. Trigger Judgment: Determine whether to trigger incremental calculation based on the feature threshold trigger condition, and call the LSTM prediction model to determine whether prediction trigger is needed;

[0037] B5. Granularity Adjustment: If incremental calculation is triggered, calculate the current granularity coefficient. Adjust the incremental calculation window ;

[0038] B6. Incremental Calculation: Window Based on Feature Port Incremental calculations are performed on the data, updating only the data items that have changed compared to the previous window, and the incremental results are stored.

[0039] B7. Loop: Enter the next time window and repeat B2-B6.

[0040] Furthermore, in step S4, the node load value L is calculated based on the incremental calculation results using a load assessment model, and the load assessment model includes CPU utilization. Memory usage and incremental data processing volume The three load assessment metrics are as follows:

[0041] CPU utilization This represents the real-time CPU utilization of the node, and its value ranges from [value range missing]. ;

[0042] Memory usage This is expressed as the real-time memory utilization rate of the node, with a value range of [value range missing]. ;

[0043] Incremental data processing volume This represents the incremental data volume that a node needs to process within the current time window, calculated using the following formula: ,in The size of the dataset for the incremental computation results. The data compression factor;

[0044] The formula for calculating the node load value L is: ,in This represents the maximum incremental data processing volume across all nodes. The range of values ​​is , The closer it is to 1, the higher the node load.

[0045] Furthermore, the strategy for predicting load and issuing early warnings using the XGBoost model in S4 is as follows:

[0046] A node prediction model is built based on the XGBoost model, i.e., gradient boosting tree, with load data from the past 15 time windows as input. and corresponding streaming data characteristics Predict the node load values ​​for the next two windows. Set load warning threshold If prediction If so, the node is marked as a "node to be scheduled", triggering the dynamic load balancing process.

[0047] Furthermore, in S4, an association-priority - minimum cost load migration strategy is executed for the nodes to be scheduled, specifically as follows:

[0048] C1. Load Collection: Collect the current load value of all queried nodes. and predicted load values Filter out nodes to be scheduled and idle nodes, among which This is an idle node;

[0049] C2. Data Association Analysis: Based on the data association relationships in the incremental calculation results, the incremental data on the nodes to be scheduled are grouped from high to low according to the degree of association, so that the data in the same association group are migrated to the same idle node;

[0050] C3. Migration Cost Calculation: Migration Cost The calculation formula is: ,in, For data migration time, This refers to the network bandwidth between nodes. To measure the impact of migration on node load, a value is set to... If the node to be scheduled after migration And idle nodes ,but This indicates minimal impact;

[0051] C4. Migration Decision: For each node to be scheduled, calculate its migration cost with all idle nodes. Choose migration cost The smallest idle node is selected as the target node for load migration.

[0052] C5. Migration Verification: After the migration is completed, collect the actual load values ​​of the node to be scheduled and the target node. If both meet the requirements... If the migration is successful, then C2-C4 will be executed again; otherwise, C2-C4 will be executed again.

[0053] Furthermore, the incremental computation and dynamic load balancing streaming data query optimization method can be applied to streaming data query optimization in financial transactions, IoT sensing, and real-time monitoring, with adaptive adjustment of feature weights and trigger thresholds in various scenarios.

[0054] In summary, this application includes at least one of the following beneficial technical effects:

[0055] Multi-dimensional feature-driven incremental computation optimization: By constructing a feature model that includes data generation frequency, fluctuation range, query relevance and priority, combined with dynamic granularity adjustment and dual triggering mechanism, the incremental computation duplication rate is effectively reduced, the query response time is significantly shortened, and the real-time query requirements of streaming data are met.

[0056] Dynamic load balancing with incremental results linkage: Based on the incremental data processing volume, a load assessment model is built. Combined with XGBoost load prediction and the "association priority - minimum cost" migration strategy, the load can be accurately predicted and efficiently scheduled, reducing node load differences and improving resource utilization.

[0057] Highly adaptable to various business scenarios: Feature weights, trigger thresholds, model parameters, etc., can all be flexibly adjusted according to different business scenarios such as finance, IoT, and monitoring, and have broad application value;

[0058] High stability and reliability: Through mechanisms such as load prediction and early warning, and migration verification, system instability caused by load adjustment delays or migration failures is avoided, ensuring the continuity and reliability of streaming data processing. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a flowchart illustrating the incremental calculation and dynamic load balancing streaming data query optimization method of this application.

[0061] Figure 2 This is a flowchart of the analytic hierarchy process in S1 of the incremental calculus and dynamic load balancing streaming data query optimization method of this application.

[0062] Figure 3 This is a flowchart illustrating the process of executing the association-first, minimum-cost load migration strategy for the node to be scheduled in S4 of the incremental calculus and dynamic load balancing streaming data query optimization method of this application. Detailed Implementation

[0063] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0064] Example 1

[0065] Please see Figures 1-3 An incremental calculus and dynamic load balancing method for optimizing streaming data queries, the control method includes:

[0066] S1. Construct a multi-dimensional streaming data feature model and determine the weights of each feature dimension;

[0067] S1 defines each feature dimension to address the dynamic characteristics of streaming data. The feature dimensions are as follows:

[0068] Data generation frequency : This represents the number of data entries generated per unit of time, calculated using the following formula: ,in, For time window Number of data stream entries within;

[0069] Data value fluctuation range Time window The maximum deviation rate of the internal data values ​​is calculated using the following formula: ,in, For the first in the window The value of each data item This represents the average value of the data within the window.

[0070] Query relevance The percentage of times this data stream is associated with historical query statements is calculated using the following formula: ,in, This represents the number of times the data stream has been associated with query statements in the past hour. This represents the total number of all query statements in the past hour.

[0071] Data priority P: The weight value is set according to business requirements, and the value range is [1,5], where 5 represents the highest priority and 1 represents the lowest priority. The priority decreases one by one from 5 to 1.

[0072] In S1, the Analytic Hierarchy Process (AHP) is used to determine the weights of each feature dimension, specifically:

[0073] A1. Constructing the judgment matrix: Five experts in the field of data processing were invited to compare the importance of the four feature dimensions pairwise. A judgment matrix was constructed using the 1-9 scale method, where 1 represents equal importance and 9 represents extreme importance, with importance increasing sequentially from 1 to 9.

[0074] A2. Consistency Test: Calculate the largest eigenvalue of the judgment matrix. And calculate the consistency index. ,in, Let be the number of feature dimensions, if If so, then the matrix satisfies the consistency requirement;

[0075] A3. Weight Determination: Normalize the eigenvectors of the judgment matrix to obtain the weights for each feature dimension. ,in, Data generation frequency Data value fluctuation range Query relevance And the weights normalized to the data priority P.

[0076] S2. Collect initial stream data, calculate the incremental calculation granularity coefficient, and determine the initial incremental window;

[0077] In S2, based on the calculation results of the multi-dimensional streaming data feature model, the incremental calculation granularity coefficient is defined. The calculation formula is: Among them, the incremental calculation granularity coefficient The range of values ​​is This corresponds to 5 incremental calculation granularity levels, and the 5 incremental calculation granularity levels are as follows:

[0078] Level 1 Coarse-grained incremental calculation uses a 10-minute time window and only updates the statistical results of the data within the window.

[0079] Level 2 : Coarse-grained incremental calculation, using a 5-minute time window, incrementally updating the statistical results and key data items within the window;

[0080] Level 3 Medium-granularity incremental calculation uses a 2-minute time window to incrementally update all data items within the window;

[0081] Level 4 : Fine-grained incremental calculation, using a 1-minute time window, incrementally updating all data items and data relationships within the window;

[0082] Level 4 Fine-grained incremental calculation uses a 30-second time window to update the data in real time within the window, while retaining historical version data.

[0083] S3. Execute the incremental calculation algorithm: collect streaming data in real time, update feature values, trigger incremental calculation through the "feature threshold + prediction trigger" mechanism, adjust the incremental window and complete the incremental calculation;

[0084] The strategy of the "feature threshold + prediction trigger" mechanism in S3 is as follows:

[0085] Feature threshold triggering strategy: Incremental update is triggered when the change in a certain feature dimension exceeds a preset threshold;

[0086] Prediction triggering strategy: Construct a streaming data feature prediction model based on Long Short-Term Memory (LSTM) networks, and input feature data from the past 10 time windows. It predicts the feature values ​​for the next window. If the prediction results show that a certain feature will exceed the threshold, it triggers incremental calculation in advance.

[0087] The specific strategy of the incremental calculus algorithm in S3 is as follows:

[0088] B1. Initialization: Obtaining the initial characteristic values ​​of the streaming data Calculate the initial particle size coefficient Determine the initial incremental calculation window ;

[0089] B2. Data Acquisition, within the Time Window Internally collected streaming data, recording the number of data entries. Data values and the number of related queries ;

[0090] B3. Feature update: Calculate the feature values ​​for the current window. ,like If there are no business adjustments, the status quo will remain unchanged.

[0091] B4. Trigger Judgment: Determine whether to trigger incremental calculation based on the feature threshold trigger condition, and call the LSTM prediction model to determine whether prediction trigger is needed;

[0092] B5. Granularity Adjustment: If incremental calculation is triggered, calculate the current granularity coefficient. Adjust the incremental calculation window ;

[0093] B6. Incremental Calculation: Window Based on Feature Port Incremental calculations are performed on the data, updating only the data items that have changed compared to the previous window, and the incremental results are stored.

[0094] B7. Loop: Enter the next time window and repeat B2-B6.

[0095] S4. Execute dynamic load balancing algorithm: Calculate node load values ​​based on incremental calculation results, predict load through XGBoost model, and perform load migration for nodes that exceed the warning threshold according to the "association priority - minimum cost" strategy.

[0096] In S4, the node load value L is calculated based on the incremental calculation results using a load assessment model, which includes CPU utilization. Memory usage and incremental data processing volume The three load assessment metrics are as follows:

[0097] CPU utilization This represents the real-time CPU utilization of the node, and its value ranges from [value range missing]. ;

[0098] Memory usage This is expressed as the real-time memory utilization rate of the node, with a value range of [value range missing]. ;

[0099] Incremental data processing volume This represents the incremental data volume that a node needs to process within the current time window, calculated using the following formula: ,in The size of the dataset for the incremental computation results. The data compression factor;

[0100] The formula for calculating the node load value L is: ,in This represents the maximum incremental data processing volume across all nodes. The range of values ​​is , The closer it is to 1, the higher the node load.

[0101] The strategy for predicting load and issuing early warnings using the XGBoost model in S4 is as follows:

[0102] A node prediction model is built based on the XGBoost model, i.e., gradient boosting tree, with load data from the past 15 time windows as input. and corresponding streaming data characteristics Predict the node load values ​​for the next two windows. Set load warning threshold If prediction If so, the node is marked as a "node to be scheduled", triggering the dynamic load balancing process.

[0103] Furthermore, in S4, an association-priority - minimum cost load migration strategy is executed for the nodes to be scheduled, specifically as follows:

[0104] C1. Load Collection: Collect the current load value of all queried nodes. and predicted load values Filter out nodes to be scheduled and idle nodes, among which This is an idle node;

[0105] C2. Data Association Analysis: Based on the data association relationships in the incremental calculation results, the incremental data on the nodes to be scheduled are grouped from high to low according to the degree of association, so that the data in the same association group are migrated to the same idle node;

[0106] C3. Migration Cost Calculation: Migration Cost The calculation formula is: ,in, For data migration time, This refers to the network bandwidth between nodes. To measure the impact of migration on node load, a value is set to... If the node to be scheduled after migration And idle nodes ,but This indicates minimal impact;

[0107] C4. Migration Decision: For each node to be scheduled, calculate its migration cost with all idle nodes. Choose migration cost The smallest idle node is selected as the target node for load migration.

[0108] C5. Migration Verification: After the migration is completed, collect the actual load values ​​of the node to be scheduled and the target node. If both meet the requirements... If the migration is successful, then C2-C4 will be executed again; otherwise, C2-C4 will be executed again.

[0109] S5. Receive user query requests, prioritize retrieving data from incremental calculation results, and if historical data is required, associate it with the stored complete data and return the query results;

[0110] S6. Monitor streaming data characteristics and node load in real time, and repeatedly execute S3-S5 to optimize query efficiency.

[0111] The incremental calculus and dynamic load balancing streaming data query optimization method can be applied to streaming data query optimization in financial transactions, IoT sensing, and real-time monitoring, with adaptive adjustment of feature weights and trigger thresholds in various scenarios.

[0112] Example 2

[0113] To verify the effectiveness of the present invention, a specific implementation is carried out using a financial transaction flow data processing scenario as an example.

[0114] Scene parameter settings;

[0115] Stream data type: Stock trading data, including fields such as stock code, trading time, transaction price, and transaction quantity;

[0116] Data scale: 1000 data points are generated per second at peak times, and 500 data points per second on average;

[0117] Query requirements: Real-time query of the average transaction price and highest transaction price of a stock within the last 5 minutes, with a query response time requirement of <1 second;

[0118] Node configuration: Five query nodes are used, each with 8 CPU cores, 16GB of memory, and 100MB / s of network bandwidth between nodes.

[0119] Model and algorithm parameter configuration

[0120] Multidimensional feature weights ;

[0121] Incremental trigger thresholds: ΔF > 20%, ΔV > 15%;

[0122] LSTM prediction model: Input layer dimension is 4 (4 features), hidden layer has 2 layers, each layer has 32 neurons, output layer dimension is 4, training data is streaming data features from the past 7 days;

[0123] Load warning threshold: ;

[0124] XGBoost load prediction model: Input layer dimension is 7 (3 load metrics + 4 streaming data features), number of trees is 100, maximum tree depth is 5, training data consists of node load and streaming data features from the past 7 days. Implementation process and results;

[0125] Model initialization: After system startup, initialize the multi-dimensional feature model and calculate the initial data generation frequency. Messages / minute (Since the average is 500 messages per second, 1 minute is 30,000 messages. This is corrected here: The average is 500 messages per second, 1 minute is 500 × 60 = 30,000 messages, therefore...) (data per minute), initial data value fluctuation range (Stock prices fluctuate relatively little in the short term), initial query relevance (This is a popular stock with a high number of related queries), data priority. (Financial transaction data has the highest priority), calculate the initial granularity coefficient. (Standardization process here: F is based on 10,000 messages / minute, V on 20%, and R on 0.5). Calculated... This corresponds to incremental calculation granularity level 1, initial window. minute;

[0126] Incremental calculus execution: Within the first 10-minute window, collect 300,000 data entries (500 entries / second × 600 seconds), and calculate the current... strips / minute (no change) (Increased price volatility), ΔV = 3% < 15%, (Increased query count), ΔR=0.2, the change in features calculated based on weights did not reach the threshold;

[0127] Meanwhile, the LSTM model predicts the next window. The value will increase to 18%, ΔV = 10% < 15%, therefore incremental calculation will not be triggered; within the second window, Increased to 19%, ΔV=11%<15%, LSTM predicts the next window It will increase to 22%, ΔV=14%<15%, still not triggered; within the third window Increased to 23%, ΔV=15%, triggering incremental calculation. It remains at level 1, and the window remains open for 10 minutes; if at some point, the stock price suddenly fluctuates... When the value increases to 40%, ΔV = 17% > 15%, triggering incremental calculation immediately. It remains within the Level 1 range, but considering the significantly increased price volatility, the LSTM model predicts the next window. This will be further increased to 50%, so the incremental calculation window will be adjusted to 8 minutes in advance (between the 10 minutes of Level 1 and the 5 minutes of Level 2, to achieve flexible granularity adjustment), and incremental updates will only be performed on transaction data with price fluctuations exceeding 40%, reducing the amount of redundant calculations;

[0128] Dynamic load balancing execution: During periods of sudden stock price fluctuations, the incremental data processing volume D of node 1 increases to 80MB (due to increased incremental update frequency), CPU utilization U=0.75, memory usage M=0.65, and the node load value L=0.4×0.75+0.3×0.65+0.3×(80 / 80)=0.3+0.195+0.3=0.795>0.7, thus it is marked as a node to be scheduled; at the same time, the load of other nodes is collected, and node 3's L=0.25 (U=0.2, M=0.2, D=20MB), indicating it is an idle node;

[0129] Data correlation analysis: The incremental data on node 1 is grouped according to the query correlation degree R. The data volume of the "Real-time transaction price query of stock A" group accounts for 60%, and the data volume of the "5-minute K-line query of stock A" group accounts for 40%. The two groups of data are packaged separately.

[0130] Migration cost calculation: Inter-node network bandwidth B = 100MB / s, total migrated data D = 80MB, therefore Second;

[0131] The predicted value of node 1 after migration is L = 0.4 × 0.5 + 0.3 × 0.45 + 0.3 × (20 / 80) = 0.2 + 0.135 + 0.075 = 0.41.

[0132] For node 3, L = 0.4 × 0.4 + 0.3 × 0.35 + 0.3 × (80 / 80) = 0.16 + 0.105 + 0.3 = 0.565, all of which satisfy L < 0.6. Therefore... ;

[0133] The migration cost C = 0.6 × 0.8 + 0.4 × 0.1 = 0.48 + 0.04 = 0.52;

[0134] Migration execution and verification: The two sets of related data were migrated to node 3. After the migration was completed, the actual test results showed that L=0.42 for node 1 and L=0.57 for node 3, both of which met the requirements, and the migration was successful.

[0135] Query response verification: In the above scenario, for the query of "average transaction price of stock A within 5 minutes", the response time using the method of this invention is 0.3 seconds, which is much lower than the response time of the traditional fixed window incremental calculation + static hash load balancing method. This invention shortens the query response time; at the same time, the average load difference of nodes is reduced from 0.4 in the traditional method to 0.15, and the resource utilization rate is improved by 62.5%.

[0136] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for query optimization of streaming data with incremental computation and dynamic load balancing, characterized in that, The control method comprises: S1. Constructing a multi-dimensional flow data feature model, determining the weight of each feature dimension; S2. Collecting initial flow data, calculating the incremental calculation granularity coefficient, and determining the initial incremental window; S3. Perform incremental calculation algorithm: real-time collection of flow data, update of feature value, triggering of incremental calculation through "feature threshold + prediction trigger" mechanism, adjustment of incremental window and completion of incremental calculation; S4. Perform dynamic load balancing algorithm: calculate node load value based on incremental calculation result, predict load through XGBoost model, and perform load migration on nodes exceeding warning threshold according to "correlation priority - minimum cost" strategy; S5. Receive user query request, preferentially obtain data from incremental calculation result, if historical data is required, associate stored complete data, and return query result; S6. Real-time monitoring of flow data features and node load, and cyclic execution of S3-S5 to optimize query efficiency.

2. The method of claim 1, wherein, In S1, each feature dimension is defined according to the dynamic characteristics of the flow data, and each feature dimension is: Data generation frequency : the number of data generated in a unit time, the calculation formula is: , wherein, is the number of data streams in the time window ; Data value fluctuation range Time window The maximum deviation rate of the internal data values ​​is calculated using the following formula: ,in, For the first in the window The value of each data item This represents the average value of the data within the window. Query correlation degree : The proportion of the number of times of association of the data stream with historical query statements, the calculation formula is: , wherein, is the number of times of association of the data stream with query statements in the past 1 hour, is the total number of query statements in the past 1 hour; Data priority P: weight value set according to business requirements, value range [1, 5], wherein 5 represents the highest priority and 1 represents the lowest priority, and the priority decreases by one from 5 to 1.

3. The method of claim 2, wherein, In S1, the weight of each feature dimension is determined by AHP, specifically: A1. Construct a judgment matrix: invite five experts in the field of flow data processing to compare the importance of four feature dimensions pairwise, and construct a judgment matrix using 1-9 scale method, wherein 1 represents equal importance and 9 represents extreme importance, and the importance increases from 1 to 9. A2. Consistency check: calculate the maximum eigenvalue of the judgment matrix and calculate the consistency index wherein, is the characteristic dimension number, if then the judgment matrix meets the consistency requirement; A3. Weight determination: normalize the eigenvectors of the judgment matrix to obtain the weight of each characteristic dimension wherein, are the normalized weights of the data generation frequency , data value fluctuation amplitude , query correlation degree and data priority P, respectively.

4. The method of claim 3, wherein, The calculation result in S2 according to the multi-dimensional flow data feature model defines an incremental calculation granularity coefficient , and the calculation formula is: , wherein the value range of the incremental calculation granularity coefficient is , corresponding to five incremental calculation granularity levels, and the five incremental calculation granularity levels are respectively: Level 1 is : Coarse-grained delta calculation, using a 10-minute time window, only the statistical results of the data in the window are updated incrementally; Level 2, i.e. : Coarser granularity delta calculation, using a 5 minute time window, with delta updates on statistical results and key data items within the window. Grade 3 is : Medium-granularity delta calculation, with a 2-minute time window, delta updates for all data items within the window; Grade 4, i.e. : Fine granularity incremental calculation, using 1 minute time window, all data items and data relationships in the window are updated incrementally; Grade 4, i.e. Fine-grained incremental computation with 30 seconds time window, real-time incremental update on data within the window, and historical version data preserved.

5. The method of claim 4, wherein, In S3, the strategy of "feature threshold + prediction trigger" mechanism is: Feature threshold trigger strategy: when the change of a feature dimension exceeds the preset threshold, trigger incremental update; Prediction trigger strategy: build a stream data feature prediction model based on long short-term memory network (LSTM), input the feature data of the past 10 time windows, predict the feature value of the future 1 window, and if the prediction result shows that a certain feature will exceed the threshold, trigger the incremental calculation in advance. , predict the feature value of the future 1 window, and if the prediction result shows that a certain feature will exceed the threshold, trigger the incremental calculation in advance.

6. The method of claim 5, wherein, In S3, the specific strategy of incremental calculation algorithm is: B1. Initialization: Obtain initial eigenvalues of the stream data , calculate initial granularity coefficients , determine initial increment calculation window ; B2. Data collection, in a time window of stream data, number of records , data values and number of query associations ; B3. Feature update, compute the pair feature value of the current window , if no service adjustment, keep it unchanged; B4. Trigger judgment: determine whether to trigger incremental calculation according to the feature threshold trigger condition, and call the LSTM prediction model to determine whether to trigger prediction; B5. Granularity adjustment: if the delta calculation is triggered, calculate the current granularity coefficient , adjust the delta calculation window ; B6. Incremental computation: Window based on feature mouth Incremental computation on data, updating only data items that changed compared to the previous window and storing the incremental result B7. Cycle: enter the next time window and repeat B2-B6.

7. The method of claim 6, wherein, The node load value L in the S4 is calculated based on the incremental calculation result, adopts a load evaluation model, and the load evaluation model includes three load evaluation indexes of CPU utilization rate , memory occupation rate and incremental data processing amount . CPU utilization Real-time usage of the CPU expressed as a node CPU, and taking values in the range ; Memory occupancy The real-time occupancy of the node memory, with a value range of ; Incremental data processing amount represents the incremental data amount that the node needs to process in the current time window, and the calculation formula is as follows: wherein is the data set size of the incremental calculation result, is a data compression coefficient; The calculation formula of the node load value L is: wherein is the maximum value of the incremental data processing amount in all nodes, The value range of is , The closer to 1 indicates that the node load is higher.

8. The method of claim 7, wherein, In S4, the strategy of predicting load and warning through XGBoost model is: Based on XGBoost model, i.e. gradient boosting tree, a node prediction model is constructed, and the load data of the past 15 time windows is input And the corresponding flow data characteristics , predict the node load value of the next 2 windows , set the load warning threshold If the prediction , mark the node as "to-be-scheduled node" and trigger the dynamic load balancing process.

9. The method of claim 8, wherein, In S4, the load migration strategy of correlation priority - minimum cost for the scheduled nodes is as follows: C1. Load collection: collect the current load values of all query nodes and predicted load values , filter out the to-be-scheduled nodes and idle nodes, wherein is an idle node; C2. Data correlation analysis: based on the data correlation relationship in the incremental calculation result, group the incremental data on the scheduled nodes according to the correlation degree from high to low, and migrate the data in the same correlation group to the same idle node; C3. Migration Cost Calculation: Migration Cost The calculation formula is: ,in, For data migration time, This refers to the network bandwidth between nodes. To measure the impact of migration on node load, a value is set to... If the node to be scheduled after migration And idle nodes ,but This indicates minimal impact; C4. Migration decision: for each node to be scheduled, calculate its migration cost with all free nodes , select the free node with the smallest migration cost as the target node, and perform load migration; ​ C5. Migration verification: After migration is completed, the actual load values of the node to be dispatched and the target node are collected. If both satisfy , the migration is successful; otherwise, C2-C4 is re-executed.

10. The method of claim 1-9, wherein, It can be applied to flow data query optimization in financial transactions, Internet of Things sensing and real-time monitoring fields, and the feature weight and trigger threshold in each scene can be adjusted adaptively.

Images