A distributed real-time data stream processing system and method

Abstract

The present invention relates to the field of real-time data processing technology, specifically a distributed real-time data stream processing system and method, the system includes a state mapping modeling module, a load dynamic calculation module, a node task adjustment module, and a resource scheduling execution module. In the present invention, by analyzing the data flow change sequence, identifying the frequency and trend of state switching, realizing dynamic identification and classification of flow states, improving the monitoring and understanding capabilities of real-time data, accurately quantifying node resource pressure in resource occupancy and load range analysis, enhancing the predictability of load balancing, providing a scientific basis for resource scheduling, combining task priority sorting and dynamic task adjustment, optimizing node computing resource utilization and task processing response speed, realizing flexible configuration of memory and computing resources, alleviating peak load pressure, improving data processing efficiency and system stability, and improving the system's adaptability and reliability in processing large-scale data flows.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time data processing, and in particular to a distributed real-time data stream processing system and method. Background Art

[0002] The field of real-time data processing technology focuses on how to efficiently and accurately process the continuously generated real-time data streams. This field involves the rapid processing, analysis and storage of real-time data through distributed computing architecture, parallel processing, low-latency transmission and other technologies. Real-time data processing systems need to have the characteristics of rapid response, dynamic adaptation and high reliability. They are widely used in financial transactions, Internet of Things monitoring, social media analysis, intelligent manufacturing, traffic management and other scenarios. The core challenges of the technology include how to process large-scale data streams, ensure high scalability and low latency of the system, and how to efficiently use computing resources.

[0003] Among them, the distributed real-time data stream processing system refers to a system that uses a distributed computing architecture to process real-time data streams. Its main purpose is to cope with scenarios that require instant processing of large amounts of real-time data, such as real-time monitoring, online transactions, data stream analysis, etc. The system distributes data to multiple nodes to ensure that the data processing speed and the system response time meet the real-time requirements. Through effective load balancing, flow control mechanisms and fault-tolerant mechanisms, it can efficiently process data streams in a large-scale distributed environment and provide fast and accurate analysis results.

[0004] The existing technology has limited state recognition capabilities in real-time data stream processing, and it is difficult to accurately analyze the state switching trends and correlation characteristics in traffic changes, resulting in insufficient adaptability to complex data scenarios. In terms of load prediction, it relies on static or historical data and cannot flexibly respond to real-time dynamic changes. It is easy to have resource allocation imbalance under high load. The node task allocation strategy relies too much on preset rules and does not fully consider the dynamic changes of real-time task priorities and computing capacity, resulting in low task allocation efficiency. In resource scheduling, the existing technology is not sensitive enough to the dynamic adjustment of the release speed and remaining amount of node resources. The utilization rate of memory and computing resources is not high, and it is easy to have resource waste or processing bottlenecks during data flow peaks, resulting in a decrease in the system's response speed when processing large-scale, high-concurrency data streams, affecting real-time and stability. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings in the prior art and to propose a distributed real-time data stream processing system and method.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: A distributed real-time data stream processing system comprises:

[0007] The state mapping modeling module analyzes the associated frequency of state switching based on the change sequence of the original data traffic record, identifies the switching trend and impact range, classifies and aggregates the switching behavior of the traffic state, and generates a data traffic state mapping diagram;

[0008] The load dynamic calculation module extracts the resource occupancy and load range of each state based on the data flow state map, analyzes the resource pressure of the node in each state, generates a node load analysis diagram, accumulates the load trend in the future time period, identifies the load regional distribution and fluctuation range, and generates a node load prediction analysis result;

[0009] The node task adjustment module analyzes the task execution pressure and unassigned task data based on the node load prediction analysis results, evaluates the execution priority and remaining computing capacity of the task, generates a task priority sorting table, adjusts the task allocation of the node, and allocates the tasks of the node to generate a task adjustment allocation table;

[0010] The resource scheduling execution module adjusts the allocation table based on the task, analyzes the remaining amount and release speed of the node resources, adjusts the allocation strategy of memory and computing resources in combination with the dynamic change data of the data flow resources, and makes plan adjustments according to the execution load and resource configuration data of the real-time node to generate a distributed data flow resource configuration plan.

[0011] As a further solution of the present invention, the step of obtaining the data flow state map is specifically as follows:

[0012] Based on the change sequence of the original data traffic record, the time series parameters and state marking parameters are called to analyze the state switching frequency of adjacent records in the time series. By comparing the change trend of the switching frequency, the key state switching behavior is identified and the preliminary classification result of the state switching behavior is generated.

[0013] Based on the preliminary classification results of the state switching behavior, the frequency distribution of each type of state switching behavior and the impact range of the overall data traffic are analyzed, using the formula:

[0014] ;

[0015] Calculate the influence coefficient of the state switching behavior and generate the classification results of the influence range;

[0016] in, Represents the influence coefficient of state switching behavior, represents the frequency of state switching behavior, The weight parameter representing the impact of switching behavior on traffic, Represents the total number of records;

[0017] In combination with the classification result of the impact range, by aggregating the classification data of the differentiated state switching behavior and utilizing the weighted analysis of the switching frequency and the impact range, a data traffic state map is generated.

[0018] As a further solution of the present invention, the step of obtaining the node load analysis diagram is specifically as follows:

[0019] According to the data traffic state map, the resource occupancy parameters and load range parameters of each state node are extracted, and the resource pressure parameters of memory occupancy, CPU occupancy, and network bandwidth usage of each state node are counted respectively. The resource consumption and load range of each state are quantified in sections through the associated parameters, and a resource load data model of the state node is established;

[0020] Based on the resource load data model of the state node, a weighted analysis is performed on the resource consumption parameters of each node using the formula:

[0021] ;

[0022] Calculate the resource pressure value of each node and generate node resource pressure analysis results;

[0023] in, Represents the resource pressure value of the node, Represents the node memory consumption, Represents the node CPU consumption, Represents the node bandwidth usage, Represents the weight factor of the resource, is the total number of states;

[0024] Combined with the node resource pressure analysis results, a time series analysis is performed on the resource pressure of each node in a differentiated state to determine the regularity of load changes, a dynamic balance analysis is performed on the loads between nodes, and a node load analysis graph is generated.

[0025] As a further solution of the present invention, the step of obtaining the node load prediction analysis result is specifically:

[0026] According to the node load analysis diagram, extract the node load data at the original time point, filter the usage and activity records on weekdays, exclude outliers and duplicate records, standardize the data format, and obtain a time series data set;

[0027] Using the time series data set, geocoding the load data of each node, combining the physical location, projecting the data onto a map using standard GIS, assigning differentiated color and size marks to each node, and obtaining node load geographic location mapping data;

[0028] According to the node load geographic location mapping data, a time series analysis is performed on the existing data to predict the load peak and fluctuation of the node in the future time period, adjust the time window and step size of the prediction model, optimize the time granularity of data presentation, and obtain the node load prediction analysis results.

[0029] As a further solution of the present invention, the step of obtaining the task priority ranking table is specifically as follows:

[0030] The node load prediction analysis results are analyzed, task execution pressure parameters and unassigned task volume parameters are extracted, the current load ratio and unassigned task ratio of each task are analyzed, and the task priority factor is preliminarily evaluated using the formula:

[0031] ;

[0032] Generate a preliminary assessment of task priority;

[0033] in, is the initial assessment value of the task priority, is the current load ratio of the task, The remaining calculation capacity of the system is is the proportion of unassigned tasks, is the weight coefficient;

[0034] Calling the parameters in the preliminary evaluation value of the task priority, combining the original data of the task execution time and the original data of the node computing capacity, calculating the dynamic priority adjustment factor of the task, correcting the task priority by the dynamic adjustment factor, and generating a task priority correction value set;

[0035] The task priority correction value set and the node remaining computing capacity are called to sort the selected task execution priorities, the priority sorting is verified by using the threshold determination logic, and a task priority sorting table is generated.

[0036] As a further solution of the present invention, the step of obtaining the task adjustment allocation table is specifically as follows:

[0037] Arrange the data in the task priority sorting table, check the current task allocation and resource usage of each task node, capture the parameters of CPU utilization, memory usage and network traffic of each node, arrange and sort the task allocation data, and obtain a task resource usage detailed table;

[0038] Using the task resource usage list, analyze the resource usage rate and task execution priority of the node, identify nodes with resource overload and long task waiting time, analyze the matching degree between each task and available resources, optimize the workload of the node, and build a node task load planning table;

[0039] Through the node task load planning table, the task execution order and resource allocation details of the node are refined, the start time and resource allocation parameters of the task are adjusted, the execution order between tasks is optimized, and the task adjustment allocation table is generated.

[0040] As a further solution of the present invention, the steps of obtaining the distributed data stream resource configuration solution are specifically as follows:

[0041] Parsing the node resource information in the task adjustment allocation table, analyzing the remaining amount and release speed of each node resource, extracting the remaining resource information, and obtaining the node resource summary information;

[0042] According to the node resource summary information and the dynamic change value of the data flow resource, the formula is adopted:

[0043] ;

[0044] Calculate resource adjustment requirement values ​​and update resource allocation parameters to obtain adjusted node resource allocation information;

[0045] in, Indicates The remaining resources of the nodes, Indicates The weight factor of the node, Indicates the resource release speed. Indicates a dynamically changing value. Adjust demand values ​​for resources;

[0046] Based on the adjusted node resource allocation information, combined with real-time node execution load data, the difference between the execution load and the adjustment demand is analyzed, the plan is adjusted and the resource configuration is optimized, and a distributed data flow resource configuration plan is generated.

[0047] A distributed real-time data stream processing method is implemented based on the above-mentioned distributed real-time data stream processing system, and comprises the following steps:

[0048] S1: Based on the change sequence of the original data traffic record, extract the continuously changing values ​​and count the time difference, analyze the state switching frequency and switching direction, analyze the switching rules and mark the switching interval of the time axis, identify the high and low peak states, map the state association to the time dimension, and generate the data traffic state switching distribution diagram;

[0049] S2: Based on the data traffic state switching distribution diagram, extract the resource occupancy and load intensity of each state, analyze the resource bearing pressure of the node in the differentiated state, divide the pressure interval and accumulate the fluctuation range of the node in the time period, combine the mean and variance of the node load distribution, and generate a node resource pressure distribution table;

[0050] S3: Based on the node resource pressure distribution table, the ratio of the number of unassigned tasks to the resource demand of each node is analyzed, the resource occupancy ratio and the computing capacity of the current node are analyzed, the priority is calculated in combination with the task execution pressure, the priority resources are allocated and the remaining amount is analyzed, and a task priority and node matching result table is generated;

[0051] S4: Based on the task priority and node matching result table, compare the task requirements and the remaining node resources, adjust the excess tasks to a sequence with a low priority, compare the total unallocated task requirements with the remaining resources of the entire network, plan a new allocation order of tasks and nodes, and generate a node task resource allocation list;

[0052] S5: Based on the node task resource allocation list, analyze the resource release speed and current execution status of the node, analyze the change data of dynamic traffic and node load, adjust the resource allocation ratio and task processing order, analyze the load status and optimize the configuration parameters, and generate a distributed resource configuration plan.

[0053] Compared with the prior art, the advantages and positive effects of the present invention are:

[0054] In the present invention, by analyzing the change sequence of the original data flow record, identifying the associated frequency and trend of the state switching, realizing the dynamic identification and classification of the flow state, effectively improving the state monitoring and understanding depth of real-time data, accurately quantifying the resource pressure of the node in different states in the resource occupancy and load range analysis, generating future load trend predictions, not only improving the predictability of load balancing, but also providing a scientific basis for resource scheduling. Through the analysis of task execution pressure and unassigned task data, combined with task priority sorting, dynamically adjusting task allocation, significantly optimizing the utilization rate of node computing resources and the response speed of task processing, in the process of adjusting the resource allocation strategy in combination with dynamically changing data, realizing the flexible configuration of memory and computing resources, effectively alleviating the pressure brought by peak load, ensuring the efficiency of data processing and the stability of the system, this holistic resource scheduling and task optimization strategy not only improves the system's dynamic adaptability to large-scale data flow processing, but also enhances the processing efficiency and reliability of complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a system flow chart of the present invention;

[0056] Figure 2 It is a flow chart of the data flow state mapping diagram in the present invention;

[0057] Figure 3 It is a flow chart of the node load analysis diagram in the present invention;

[0058] Figure 4It is a flow chart of the node load prediction analysis results in the present invention;

[0059] Figure 5 A flowchart of a task priority ranking table in the present invention;

[0060] Figure 6 A flowchart of the task adjustment allocation table in the present invention;

[0061] Figure 7 It is a flow chart of the distributed data stream resource configuration solution in the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0064] See also Figure 1 The present invention provides a technical solution: a distributed real-time data stream processing system comprising:

[0065] The state mapping modeling module analyzes the associated frequency of state switching based on the change sequence of the original data traffic record, identifies the switching trend and impact range, classifies and aggregates the switching behavior of the traffic state, and generates a data traffic state mapping diagram;

[0066] The load dynamic calculation module extracts the resource occupancy and load range of each state based on the data traffic state map, analyzes the resource pressure of the node in each state, generates a node load analysis diagram, accumulates the load trend in the future time period, identifies the load regional distribution and fluctuation range, and generates node load prediction analysis results;

[0067] The node task adjustment module analyzes the task execution pressure and unassigned task data based on the node load prediction analysis results, evaluates the task execution priority and remaining computing capacity, generates a task priority sorting table, adjusts the task allocation of the node, and allocates the tasks of the node to generate a task adjustment allocation table;

[0068] The resource scheduling execution module adjusts the allocation table based on the task, analyzes the remaining amount and release speed of node resources, adjusts the allocation strategy of memory and computing resources based on the dynamic change data of data flow resources, and makes plan adjustments based on the execution load and resource configuration data of real-time nodes to generate a distributed data flow resource configuration plan.

[0069] The data traffic state mapping diagram includes the state switching sequence, associated frequency, and switching trend. The node load analysis diagram includes resource occupancy, load range, and resource pressure. The node load prediction analysis results include load trend, regional distribution, and fluctuation range. The task priority sorting table includes task classification, priority level, and remaining computing capacity. The task adjustment allocation table includes task execution pressure, unassigned tasks, and task execution priority. The distributed data flow resource configuration plan includes memory configuration, computing resource allocation, and load adjustment strategy.

[0070] See also Figure 2 , the steps for obtaining the data traffic status map are as follows:

[0071] Based on the change sequence of the original data traffic record, the time series parameters and state marking parameters are called to analyze the state switching frequency of adjacent records in the time series. By comparing the change trend of the switching frequency, the key state switching behavior is identified and the preliminary classification result of the state switching behavior is generated.

[0072] The time series parameters and state mark parameters in the original data traffic record are called, and the time series change characteristics of each record are extracted by parsing the timestamp and the corresponding state mark in the original data stream. The switching frequency between adjacent records is calculated using the time interval as the basic variable, and then the range and distribution characteristics of the switching frequency are classified and counted to generate a frequency distribution curve. At the same time, the time nodes of the state switching are segmented, and the concentrated area of ​​high-frequency switching behavior and the boundary area of ​​low-frequency switching behavior are analyzed. Combined with the distribution characteristics of the switching target state, the switching behavior is identified by matching the state type and time interval of each record in the time series one by one, and a preliminary switching behavior classification is established. At the same time, the correlation between the switching direction and the target state is considered, and the high-frequency switching area is further confirmed and marked to generate a preliminary classification result of the state switching behavior.

[0073] Based on the preliminary classification results of state switching behaviors, the frequency distribution of each type of state switching behavior and the impact range of the overall data traffic are analyzed using the formula:

[0074] ;

[0075] Calculate the influence coefficient of the state switching behavior and generate the classification results of the influence range;

[0076] in, Represents the influence coefficient of state switching behavior, represents the frequency of state switching behavior, The weight parameter representing the impact of switching behavior on traffic, Represents the total number of records;

[0077] The benefit of the formula is that by combining the frequency of switching behavior and the weight of its impact range, the overall impact of each type of switching behavior on data traffic can be accurately calculated, thereby optimizing the accuracy of classification and subsequent analysis;

[0078] Assume the time series contains the total number of records , the switching behavior frequency corresponding to each record is [3, 4, 2, 5, 6], weight is [0.8, 0.9, 0.7, 0.85, 0.95]. First, calculate the product of frequency and weight: ;

[0079] Then sum the values: ;

[0080] Then calculate the sum of the weights: ;

[0081] Substituting both into the formula: ;

[0082] The result shows that the influence coefficient of state switching behavior is 4.13, which indicates the overall impact range and intensity of switching behavior in data traffic state, and provides a basic basis for subsequent classification.

[0083] Combined with the classification results of the impact range, by aggregating the classification data of differentiated state switching behaviors, and using weighted analysis of switching frequency and impact range, a data traffic state map is generated;

[0084] Combined with the classification results of the impact range, by statistically analyzing the distribution range of the impact coefficient of each type of state switching behavior, the high impact coefficient category is extracted and its association characteristics with the low impact coefficient category are determined. The directionality of the switching behavior and the switching target distribution are used as basic variables, and the switching behaviors with high impact coefficients are prioritized for aggregation through a weighted analysis method. According to the differences in switching intensity between different categories, their aggregation impact on the global data traffic state is analyzed. The boundary relationship between the high impact coefficient category and the adjacent low impact coefficient category is identified through cross-validation. The boundary categories within the impact range are optimized by item-by-item aggregation. At the same time, the aggregation strategy of the state aggregation module is readjusted according to the frequency distribution and directional consistency of the switching behavior to generate a data traffic state map.

[0085] See also Figure 3 , the specific steps for obtaining the node load analysis diagram are:

[0086] According to the data traffic state map, the resource occupancy parameters and load range parameters of each state node are extracted, and the resource pressure parameters of memory occupancy, CPU occupancy, and network bandwidth usage of each state node are counted respectively. The resource consumption and load range of each state are quantified in sections through the associated parameters, and the resource load data model of the state node is established;

[0087] The resource occupancy parameters and load range parameters of each state node in the data traffic state map are extracted, and the resource occupancy records of each state are called, including parameters such as memory usage, CPU usage and network bandwidth occupancy, and the parameters are segmented and counted according to the time series records. By analyzing the resource occupancy trend in the time series, cluster analysis is performed on the occupancy values ​​of memory, CPU, network, etc. to distinguish the resource occupancy patterns of nodes in different states. At the same time, combined with the node's operating status mark, the resource consumption characteristics of the node in different states are determined, and the segmented statistical results are associated with the node mark in the time series to further quantify the load range of each state. Finally, a resource load data model for the state node is established.

[0088] Based on the resource load data model of the status node, a weighted analysis is performed on the resource consumption parameters of each node using the formula:

[0089] ;

[0090] Calculate the resource pressure value of each node and generate node resource pressure analysis results;

[0091] in, Represents the resource pressure value of the node, Represents the node memory consumption, Represents the node CPU consumption, Represents the node bandwidth usage, Represents the weight factor of the resource, is the total number of states;

[0092] The benefit of the formula is that by introducing the weighted square sum calculation of multiple resource parameters, it can comprehensively reflect the resource pressure of each node, and at the same time increase the weight of the high-occupancy parameters on the results through square processing, thereby enhancing the accuracy and sensitivity of the calculation;

[0093] Assume that the total number of states is , the memory usage of each state is [10, 15, 12, 20, 25], CPU usage is [30, 40, 35, 45, 50], bandwidth occupancy is [5, 7, 6, 8, 9], the weight factor is [0.6, 0.7, 0.8, 0.75, 0.9]. First, calculate the sum of the weighted squares of each resource parameter:

[0094] Memory section: ;

[0095] CPU part: ;

[0096] Bandwidth section: ;

[0097] Substituting the above results into the formula:

[0098] ;

[0099] The result shows that the resource pressure of the node is 35.15. This value combines the weighted occupancy of memory, CPU and bandwidth, and is an accurate quantification of the resource load of each node.

[0100] Combined with the node resource pressure analysis results, conduct a time series analysis of the resource pressure of each node under differentiated states, determine the regularity of load changes, conduct a dynamic balance analysis of the load between nodes, and generate a node load analysis graph;

[0101] First, the time period data under each state is sorted out, and the resource consumption trend in the time series is extracted to analyze the curve of node load change over time. The mean and peak value of resource consumption of nodes in each state in different time periods are further calculated, and the areas with sudden increase in resource pressure change are marked. By matching the relationship between the area and the state node in the time series, the law of resource pressure change with load change is analyzed. At the same time, combined with the node pressure distribution data during peak hours, the load balance calculation between all nodes is performed. Finally, the node pressure time series is combined with the load balancing result to generate a node load analysis diagram.

[0102] See also Figure 4 ,The specific steps for obtaining the node load prediction analysis results are:

[0103] According to the node load analysis chart, extract the node load data at the original time point, filter the usage and activity records on weekdays, exclude outliers and duplicate records, standardize the data format, and obtain the time series data set;

[0104] Parse the timestamps and load values ​​corresponding to the nodes, verify all raw data line by line to ensure that the time points match the load data correctly, and filter the records of working days according to the calendar. To avoid statistical errors, check the logical consistency of the load data, and eliminate abnormal values ​​that exceed the reasonable load range by setting upper and lower limits. For example, set the load range to 0 to 1000 units and eliminate data that exceeds this range. For the problem of duplicate records, verify based on the joint uniqueness of the timestamp and node identifier, delete duplicates and retain the first valid record. To achieve standardization, convert the time data into a standard 24-hour format, and map the load values ​​to percentage units in proportion to ensure the uniformity of subsequent processing. The processed data is organized into a time series data set, in which each record contains a clear time point, node identifier and corresponding load value to ensure the accuracy and standardization of the data, laying a data foundation for subsequent analysis.

[0105] Using time series data sets, geocode the load data of each node, combine the physical location, project the data onto the map using standard GIS, assign differentiated color and size marks to each node, and obtain node load geographic location mapping data;

[0106] Combined with the physical location parameters of each node (such as longitude, latitude and altitude information), the load data is bound to the geographic coordinates, the geographic boundaries and projection rules of the node area are confirmed, and the location of the node is mapped to a two-dimensional plane map by loading the geographic projection rules of the standard GIS tool. The visualization parameters are set according to the size of the load value, and the gradient range of the load color and the proportion of the size mark are clarified. For example, the load values ​​correspond to colors from light blue to dark red from low to high, and the load difference is presented by enlarging the symbol area. The geographic distribution map of the node not only reflects the location, but also intuitively reflects the load conditions of different nodes. During the mapping process, multiple checks are required to ensure that the projection data is consistent with the original time series data. During the processing, the node data that cannot match the location is deleted to avoid incorrect markings, and the node load geographic location mapping data containing geographic information and load visualization marks is obtained.

[0107] According to the node load geographic location mapping data, time series analysis is performed on the existing data to predict the load peak and fluctuation of the node in the future time period, adjust the time window and step size of the prediction model, optimize the time granularity of data presentation, and obtain the node load prediction analysis results;

[0108] First, the mapping data is sorted in time series to ensure that the load data of each node is arranged in chronological order, and the peak value and fluctuation range of each node are clearly marked. The data of each node is partitioned, and its fluctuation range is counted according to the daily cycle and weekly cycle respectively. The load mean and maximum value in a specific time period are extracted through a rolling window, and the impact of different time windows on fluctuations is clarified. In order to improve the analysis accuracy, the time window is gradually reduced and the step size is adjusted. The change trend of the load peak in a short time is calculated to determine a reasonable time granularity and reduce data prediction errors. By observing the trend of the time series, the node load prediction analysis results are obtained, the potential load peak period is pointed out, and the high-risk areas are marked in combination with the distribution characteristics of the geographical location to ensure that the data prediction results can accurately assist subsequent optimization and decision-making.

[0109] See also Figure 5 ,The specific steps for obtaining the task priority sorting table are:

[0110] Analyze the node load prediction analysis results, extract the task execution pressure parameters and the unassigned task volume parameters, analyze the current load ratio and the unassigned task ratio of each task, and make a preliminary evaluation of the task priority factor using the formula:

[0111] ;

[0112] Generate a preliminary assessment of task priority;

[0113] in, is the initial assessment value of the task priority, is the current load ratio of the task, The remaining calculation capacity of the system is is the proportion of unassigned tasks, is the weight coefficient;

[0114] The benefit of the formula is that by considering both the task load ratio and the unassigned task ratio, combined with the remaining computing capacity of the system, the task priority can be adjusted dynamically, thereby effectively balancing the task scheduling load and improving the node resource utilization;

[0115] Current load ratio of the task The remaining computing capacity of the system is obtained by dividing the CPU usage of the node load prediction result by the total computing capacity. The proportion of unassigned tasks is obtained by subtracting the current computing load from the total computing capacity. The weight coefficient is calculated by dividing the number of unassigned tasks by the total number of tasks. According to the statistical result of the impact of unassigned tasks on the priority of the node in the historical scheduling task, it is set to 0.6 and substituted into the formula to calculate as follows:

[0116] Known: , , , ,calculate :

[0117] ;

[0118] The result shows that the preliminary evaluation value of the task priority is 1.19. The larger the value, the more important the task is. In the subsequent steps, it will be used for further calculation and sorting, calling the parameters and historical scheduling data in the preliminary evaluation value of the task priority.

[0119] Call the parameters in the preliminary evaluation value of the task priority, combine the original data of the task execution time and the original data of the node computing capacity, calculate the dynamic priority adjustment factor of the task, correct the task priority by the dynamic adjustment factor, and generate a task priority correction value set;

[0120] Firstly, the task completion time, failure rate and fluctuation range of node computing capacity in the historical data of task execution time are extracted, the task completion time is discretized and divided into different time periods, the task failure rate of each time period is calculated, and the fluctuation range of historical node capacity is weighted summed and normalized by the weight distribution method. The above calculation results are combined with the preliminary evaluation value of task priority to calculate the dynamic priority adjustment factor of the task. The weight of the task priority in the dynamic adjustment factor is adjusted by the weight correction logic to generate a set of task priority correction values.

[0121] Call the task priority correction value set and the node remaining computing capacity to sort the selected task execution priorities, use the threshold judgment logic to verify the priority sorting, and generate a task priority sorting table;

[0122] Call the task priority correction value set and the node's remaining computing capacity, first collect real-time data on the node's remaining computing capacity and perform partition calculations, extract the available proportion of the remaining capacity, and calculate the corrected priority of each task in combination with the task priority correction value set. Arrange the task list in descending order based on the corrected priority, and use the priority threshold method to eliminate low-priority tasks to complete the rationality verification of the priority sorting. Construct a task priority sorting table based on the verified results.

[0123] See also Figure 6 , the specific steps for obtaining the task adjustment allocation table are:

[0124] Arrange the data in the task priority table, check the current task allocation and resource usage of each task node, capture the parameters of CPU utilization, memory usage and network traffic of each node, arrange and sort the task allocation data, and obtain the task resource usage details table;

[0125] First, check the current task allocation and resource usage of each task node to ensure that the task and node are correctly paired. During the data collection phase, carefully record the specific values ​​of each node's CPU utilization, memory usage, and network traffic. The parameters are obtained in real time through the on-site monitoring system. By classifying and summarizing the parameters of each node, a detailed database is created, including the average CPU utilization, the maximum and minimum memory usage, and the network peak traffic. The collected data is logically checked, such as checking and correcting data points with CPU utilization exceeding 100%. The task allocation data is systematically arranged and sorted, and the task nodes are sorted using data analysis tools. They are marked from high to low according to resource utilization. In this process, special attention is paid to the integrity and accuracy of the data to form a detailed table of task resource usage. This table not only shows the resource usage of each node, but also reflects the resource allocation of each task, providing a basis for further resource optimization.

[0126] Use the task resource usage details table to analyze the node resource usage rate and task execution priority, identify nodes with resource overload and long task waiting time, analyze the matching degree between each task and available resources, optimize the node workload, and build a node task load planning table;

[0127] By comparing the resource utilization of each node with the priority of its assigned tasks, we can identify which nodes have resource overload or long task waiting time problems, and pay special attention to those nodes whose CPU utilization is continuously higher than the average level by 80%, and nodes with abnormal surges in memory or network traffic. For nodes, we further analyze their task allocation, identify the key tasks that cause resource bottlenecks, evaluate the degree of match between each task and available resources, and make specific optimization suggestions for tasks with inappropriate resource matching, such as adjusting the resource requirements of tasks or replacing execution nodes. Through analysis, we build a node task load planning table, which not only indicates the current workload of each node, but also predicts the trend of future resource usage and provides adjustment suggestions to achieve optimal resource usage.

[0128] Through the node task load planning table, refine the node task execution order and resource allocation details, adjust the task start time and resource allocation parameters, optimize the execution order between tasks, and generate a task adjustment allocation table;

[0129] According to the urgency and resource requirements of the task, the start time of the task is adjusted to ensure that high-priority tasks can obtain the necessary resources first. For each node, the historical efficiency and resource usage of its task execution are analyzed, and the resource allocation parameters are optimized based on this information. For example, more memory resources are allocated to memory-intensive tasks, and for network-intensive tasks, their network bandwidth is guaranteed first. The execution order between tasks is adjusted to reduce resource conflicts and waiting time between tasks. For example, two tasks with high CPU usage are staggered in time to avoid resource competition caused by simultaneous operation. A task adjustment allocation table is generated, which lists in detail the new start time, resource allocation details and execution order of each task, ensuring that the resources of the entire system are effectively utilized, while improving the efficiency and response speed of task execution.

[0130] See also Figure 7 ,The specific steps for obtaining the distributed data stream resource configuration plan are:

[0131] Parse the node resource information in the task adjustment allocation table, analyze the remaining amount and release speed of each node resource, extract the remaining resource information, and obtain the node resource summary information;

[0132] Parsing the task adjustment allocation table involves an in-depth analysis of node resource information. The remaining amount of resources and the release speed of each node are key data points. The data is collected through a real-time monitoring system and updated to the database in real time. The resource status of each node is recorded in real time through sensors and monitoring software. After information collection, it will be transmitted to the central processing system through a data interface. In this system, the data will be integrated and displayed in a total resource matrix. The matrix not only contains the current resource status of each node, but also includes historical data and predicted data on resource consumption. Using this information, the system can extract the total amount of resources of all nodes and their respective remaining resource information, which is the basis for resource scheduling decisions and generates node resource summary information. The information will directly affect resource allocation strategies and decisions, ensuring the optimization of resource allocation and efficient node operation.

[0133] According to the node resource summary information and the dynamic change value of the data flow resources, the formula is adopted:

[0134] ;

[0135] Calculate resource adjustment requirement values ​​and update resource allocation parameters to obtain adjusted node resource allocation information;

[0136] in, Indicates The remaining resources of the nodes, Indicates The weight factor of the node, Indicates the resource release speed. Indicates a dynamically changing value. Adjust demand values ​​for resources;

[0137] The benefit of the formula is that it dynamically reflects the adjustment of resource release speed and change value by considering the weighted sum of the remaining amount of resources and weight factors of each node, thereby optimizing the resource allocation strategy and improving resource utilization and scheduling flexibility;

[0138] There are 3 nodes, unit, unit, Unit, representing the remaining resources of each node, and the corresponding weight factor , , , resource release speed Unit / hour, resource change value Unit, the calculation process is as follows:

[0139] ;

[0140] The result shows that the current resource adjustment requirement value is 11.84 units. This result reflects the degree to which the system needs to reallocate resources to maintain the balance and efficiency of system operation, taking into account node weights and resource changes.

[0141] Based on the adjusted node resource allocation information and combined with the real-time node execution load data, the difference between the execution load and the adjustment requirements is analyzed, the plan is adjusted and the resource allocation is optimized, and a distributed data flow resource allocation plan is generated;

[0142] According to the real-time node execution load data and the adjusted resource allocation information, the system needs to analyze the deviation between the data and the preset load target in real time. The process involves a lot of data analysis and processing. The real-time load data and resource configuration data of each node are extracted from the database. The data is obtained in real time through network monitoring and sensing technology, and is encoded into an analyzable format and stored in the central database. Using data analysis tools, the extracted data is deeply analyzed to find out the deviation value between the load and resource configuration, and adjust the resource configuration according to the deviation value. The adjustment plan will take into account the actual operating status of each node and the predicted future load, which ensures the accuracy and real-time nature of resource configuration. Finally, a distributed data stream resource configuration plan is generated, which will directly affect the performance and stability of the entire network. After optimizing the resource configuration, the node can process the data stream more effectively and improve the response speed and processing capacity of the overall network.

[0143] The distributed real-time data stream processing method is executed based on the above-mentioned distributed real-time data stream processing system, and includes the following steps:

[0144] S1: Based on the change sequence of the original data traffic record, extract the continuously changing values ​​and count the time difference, analyze the state switching frequency and switching direction, analyze the switching rules and mark the switching interval of the time axis, identify the high and low peak states, map the state association to the time dimension, and generate the data traffic state switching distribution diagram;

[0145] S2: Based on the data traffic state switching distribution diagram, extract the resource occupancy and load intensity of each state, analyze the resource bearing pressure of the node in the differentiated state, divide the pressure interval and accumulate the fluctuation range of the node in the time period, combine the mean and variance of the node load distribution, and generate the node resource pressure distribution table;

[0146] S3: Based on the node resource pressure distribution table, parse the ratio of the number of unassigned tasks to resource requirements for each node, analyze the resource occupancy ratio and the computing capacity of the current node, calculate the priority based on the task execution pressure, allocate priority resources and parse the remaining amount, and generate a task priority and node matching result table;

[0147] S4: Based on the task priority and node matching result table, compare the task requirements with the remaining node resources, adjust the excess tasks to a low-priority sequence, compare the total unassigned task requirements with the remaining resources of the entire network, plan a new assignment order for tasks and nodes, and generate a node task resource assignment list;

[0148] S5: Based on the node task resource allocation list, analyze the node's resource release speed and current execution status, analyze the dynamic traffic change data and node load, adjust the resource allocation ratio and task processing order, analyze the load status and optimize the configuration parameters, and generate a distributed resource configuration plan.

[0149] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A distributed real-time data stream processing system, characterized in that: The system comprises: The state mapping modeling module analyzes the associated frequency of state switching based on the change sequence of the original data traffic record, identifies the switching trend and impact range, classifies and aggregates the switching behavior of the traffic state, and generates a data traffic state mapping diagram; The load dynamic calculation module extracts the resource occupancy and load range of each state based on the data flow state map, analyzes the resource pressure of the node in each state, generates a node load analysis diagram, accumulates the load trend in the future time period, identifies the load regional distribution and fluctuation range, and generates a node load prediction analysis result; The node task adjustment module analyzes the task execution pressure and unassigned task data based on the node load prediction analysis results, evaluates the execution priority and remaining computing capacity of the task, generates a task priority sorting table, adjusts the task allocation of the node, and allocates the tasks of the node to generate a task adjustment allocation table; The resource scheduling execution module adjusts the allocation table based on the task, analyzes the remaining amount and release speed of the node resources, adjusts the allocation strategy of memory and computing resources in combination with the dynamic change data of the data flow resources, and makes plan adjustments according to the execution load and resource configuration data of the real-time node to generate a distributed data flow resource configuration plan.

2. The distributed real-time data stream processing system according to claim 1, characterized in that: The steps of obtaining the data flow status map are specifically as follows: Based on the change sequence of the original data traffic record, the time series parameters and state marking parameters are called to analyze the state switching frequency of adjacent records in the time series. By comparing the change trend of the switching frequency, the key state switching behavior is identified and the preliminary classification result of the state switching behavior is generated. Based on the preliminary classification results of the state switching behavior, the frequency distribution of each type of state switching behavior and the impact range of the overall data traffic are analyzed, using the formula: ； Calculate the influence coefficient of the state switching behavior and generate the classification results of the influence range; in, Represents the influence coefficient of state switching behavior, represents the frequency of state switching behavior, The weight parameter representing the impact of switching behavior on traffic, Represents the total number of records; In combination with the classification result of the impact range, by aggregating the classification data of the differentiated state switching behavior and utilizing the weighted analysis of the switching frequency and the impact range, a data traffic state map is generated.

3. The distributed real-time data stream processing system according to claim 2, characterized in that: The steps for obtaining the node load analysis diagram are specifically as follows: According to the data traffic state map, the resource occupancy parameters and load range parameters of each state node are extracted, and the resource pressure parameters of memory occupancy, CPU occupancy, and network bandwidth usage of each state node are counted respectively. The resource consumption and load range of each state are quantified in sections through the associated parameters, and a resource load data model of the state node is established; Based on the resource load data model of the state node, a weighted analysis is performed on the resource consumption parameters of each node using the formula: ； Calculate the resource pressure value of each node and generate node resource pressure analysis results; in, Represents the resource pressure value of the node, Represents the node memory consumption, Represents the node CPU consumption, Represents the node bandwidth usage, Represents the weight factor of the resource, is the total number of states; Combined with the node resource pressure analysis results, a time series analysis is performed on the resource pressure of each node in a differentiated state to determine the regularity of load changes, a dynamic balance analysis is performed on the loads between nodes, and a node load analysis graph is generated.

4. The distributed real-time data stream processing system according to claim 3, characterized in that: The steps for obtaining the node load prediction analysis results are specifically as follows: According to the node load analysis diagram, extract the node load data at the original time point, filter the usage and activity records on weekdays, exclude outliers and duplicate records, standardize the data format, and obtain a time series data set; Using the time series data set, geocoding the load data of each node, combining the physical location, projecting the data onto a map using standard GIS, assigning differentiated color and size marks to each node, and obtaining node load geographic location mapping data; According to the node load geographic location mapping data, a time series analysis is performed on the existing data to predict the load peak and fluctuation of the node in the future time period, adjust the time window and step size of the prediction model, optimize the time granularity of data presentation, and obtain the node load prediction analysis results.

5. The distributed real-time data stream processing system according to claim 4, characterized in that: The steps for obtaining the task priority ranking table are specifically as follows: The node load prediction analysis results are analyzed, task execution pressure parameters and unassigned task volume parameters are extracted, the current load ratio and unassigned task ratio of each task are analyzed, and the task priority factor is preliminarily evaluated using the formula: ； Generate a preliminary assessment of task priority; in, is the initial assessment value of the task priority, is the current load ratio of the task, The remaining calculation capacity of the system is is the proportion of unassigned tasks, is the weight coefficient; Calling the parameters in the preliminary evaluation value of the task priority, combining the original data of the task execution time and the original data of the node computing capacity, calculating the dynamic priority adjustment factor of the task, correcting the task priority by the dynamic adjustment factor, and generating a task priority correction value set; The task priority correction value set and the node remaining computing capacity are called to sort the selected task execution priorities, the priority sorting is verified by using the threshold determination logic, and a task priority sorting table is generated.

6. The distributed real-time data stream processing system according to claim 5, characterized in that: The steps for obtaining the task adjustment allocation table are specifically as follows: Arrange the data in the task priority sorting table, check the current task allocation and resource usage of each task node, capture the parameters of CPU utilization, memory usage and network traffic of each node, arrange and sort the task allocation data, and obtain a task resource usage detailed table; Using the task resource usage list, analyze the resource usage rate and task execution priority of the node, identify nodes with resource overload and long task waiting time, analyze the matching degree between each task and available resources, optimize the workload of the node, and build a node task load planning table; Through the node task load planning table, the task execution order and resource allocation details of the node are refined, the start time and resource allocation parameters of the task are adjusted, the execution order between tasks is optimized, and the task adjustment allocation table is generated.

7. The distributed real-time data stream processing system according to claim 6, characterized in that: The steps for obtaining the distributed data stream resource configuration solution are specifically as follows: Parsing the node resource information in the task adjustment allocation table, analyzing the remaining amount and release speed of each node resource, extracting the remaining resource information, and obtaining the node resource summary information; According to the node resource summary information and the dynamic change value of the data flow resource, the formula is adopted: ； Calculate resource adjustment requirement values ​​and update resource allocation parameters to obtain adjusted node resource allocation information; in, Indicates The remaining resources of the nodes, Indicates The weight factor of the node, Indicates the resource release speed. Indicates a dynamically changing value. Adjust demand values ​​for resources; Based on the adjusted node resource allocation information, combined with real-time node execution load data, the difference between the execution load and the adjustment demand is analyzed, the plan is adjusted and the resource configuration is optimized, and a distributed data stream resource configuration plan is generated.

8. A distributed real-time data stream processing method, characterized in that: The distributed real-time data stream processing system according to any one of claims 1 to 7 is implemented, comprising the following steps: Based on the change sequence of the original data traffic record, extract the continuously changing values ​​and count the time difference, analyze the state switching frequency and switching direction, analyze the switching rules and mark the switching interval of the time axis, identify the high and low peak states, map the state association to the time dimension, and generate a data traffic state switching distribution diagram; Based on the data traffic state switching distribution diagram, the resource occupancy and load intensity of each state are extracted, the resource bearing pressure of the node in the differentiated state is analyzed, the pressure interval is divided and the fluctuation range of the node in the time period is accumulated, and the node resource pressure distribution table is generated by combining the mean and variance of the node load distribution; Based on the node resource pressure distribution table, the ratio of the number of unassigned tasks to resource demand of each node is parsed, the resource occupancy ratio and the computing capacity of the current node are analyzed, the priority is calculated in combination with the task execution pressure, priority resources are allocated and the remaining amount is parsed, and a task priority and node matching result table is generated; Based on the task priority and node matching result table, compare the task requirements with the remaining node resources, adjust the excess tasks to a sequence with a low priority, compare the total unallocated task requirements with the remaining resources of the entire network, plan a new allocation order of tasks and nodes, and generate a node task resource allocation list; Based on the node task resource allocation list, analyze the node's resource release speed and current execution status, analyze the dynamic traffic change data and node load, adjust the resource allocation ratio and task processing order, analyze the load status and optimize the configuration parameters, and generate a distributed resource configuration plan.

Images