An efficient statistical processing system and method for real-time data streams

Abstract

The present invention belongs to the field of data statistics and relates to data analysis technology, aiming to solve the problems that the existing real-time data stream statistical processing system cannot flexibly adjust the shunt strategy, nor can it monitor and dynamically adjust the changes in the data stream. Specifically, it is an efficient statistical processing system and method for real-time data streams, including a statistical analysis platform, which is communicatively connected to a processing node analysis module, a data shunt module, a shunt mode testing module, and a dynamic monitoring and adjustment module; the processing node analysis module is used to analyze and evaluate the capabilities of the processing nodes of the real-time data stream, the data shunt module is used to perform shunt processing on the real-time data stream in different modes, the shunt mode testing module is used to test and analyze different shunt modes, and the dynamic monitoring and adjustment module is used to dynamically adjust the statistical processing according to the monitoring data of the real-time data stream; the present invention can flexibly select the shunt strategy and achieve the balance between processing efficiency and system stability.

Description

Technical Field

[0001] The present invention belongs to the field of data statistics, relates to data analysis technology, and specifically is an efficient statistical processing system and method for real-time data streams. Background Art

[0002] A real-time data stream is a continuous and high-speed arriving data sequence. In today's digital age, the generation and dissemination speed of data has increased exponentially. The statistical processing of real-time data streams aims to analyze continuously and rapidly arriving data and extract valuable information. The processing of real-time data streams has become crucial in many fields.

[0003] However, when dealing with high-speed, continuous, and large-scale real-time data streams, many challenges are often faced. The generation speed of real-time data streams is extremely fast, and thousands or even tens of thousands of transaction data may be generated per second. The arrival order of data may be different from the generation order, and the disorderly arriving data will increase the processing difficulty and lead to inaccurate statistical results.

[0004] Existing statistical processing systems for real-time data streams often do not fully consider the capacity differences of processing nodes, cannot flexibly adjust the shunt strategy according to the actual situation, nor can they monitor and dynamically adjust the changes in the data stream, affecting the overall processing efficiency and system performance.

[0005] In view of the above technical problems, this application proposes a solution. Summary of the Invention

[0006] The purpose of the present invention is to provide an efficient statistical processing system and method for real-time data streams, which is used to solve the problems that existing statistical processing systems for real-time data streams cannot flexibly adjust the shunt strategy according to the actual situation, nor can they monitor and dynamically adjust the changes in the data stream;

[0007] The technical problem that the present invention needs to solve is: how to provide an efficient statistical processing method and system for real-time data streams that can flexibly adjust the shunt strategy according to the actual situation and can also monitor and dynamically adjust the changes in the data stream.

[0008] The purpose of the present invention can be achieved through the following technical solutions:

[0009] An efficient statistical processing system for real-time data streams includes a statistical analysis platform. The statistical analysis platform is communicatively connected to a processing node analysis module, a data shunt module, a shunt mode test module, and a dynamic monitoring and adjustment module;

[0010] The processing node analysis module is used to analyze and evaluate the capabilities of the processing nodes of the real-time data stream: mark the nodes that receive the real-time data stream and perform statistical processing as processing nodes, and obtain the processing coefficient CL of the processing nodes;

[0011] The data shunting module is used to perform shunting processing on real-time data streams in different modes: when using processing nodes to process real-time data streams, the real-time data streams are shunted to different processing nodes; the shunting rules are based on the processing coefficient CL of the processing nodes and are divided into three modes: priority mode, balanced mode, and hybrid mode;

[0012] The shunting mode testing module is used to test and analyze different shunting modes: generate a number of test cycles Tn with a fixed duration of T seconds, where n represents the number of test cycles, n = 1, 2, ……, k, and k is a positive integer; obtain the processing mean CJ and processing floating value CF of different test cycles under different shunting modes, and determine the processing mode based on the processing mean CJ and processing floating value CF. The processing mode includes the efficiency priority mode and the stability priority mode;

[0013] Perform statistical processing on real-time data streams under different processing modes, generate a monitoring cycle with a fixed duration of L2 seconds, draw a line chart of the total data input SZ - monitoring cycle, and determine whether there is an increment in the real-time data stream, generate corresponding signals, and take corresponding measures for adjustment.

[0014] Furthermore, the process of obtaining the processing coefficient CL of the processing node includes: obtaining the processing speed CS, processing delay CY, and maximum throughput TT of the processing node, and inputting the processing speed CS, processing delay CY, and maximum throughput TT into a multi-layer perceptron model to obtain the processing coefficient CL of the processing node.

[0015] Furthermore, the processing speed CS represents the average number of bytes of data processed by the processing node within a preset time; the processing delay CY represents the average time delay from when the data enters the processing node to when the processing is completed and the result is output; the maximum throughput TT represents the maximum data flow that the processing node can process.

[0016] Furthermore, the shunting rule of the priority mode is specifically: mark the area for temporarily storing data in the processing node as a buffer, arrange the processing nodes in descending order of the processing coefficient CL to obtain a node priority sequence. When shunting the real-time data stream, give priority to the processing node with a small serial number in the node priority sequence and with remaining storage space in the buffer.

[0017] Furthermore, the shunting rule of the balanced mode is: when shunting the real-time data stream, generate a shunting cycle with a fixed duration of L1 seconds. The real-time data stream of the first shunting cycle is given to the first processing node in the node priority sequence, the real-time data stream of the second shunting cycle is given to the second processing node in the node priority sequence, ……, and so on. Until the last processing node is allocated the real-time data stream, the real-time data stream of the next shunting cycle is given to the first processing node in the node priority sequence, ……, and it goes in circles.

[0018] Further, the shunt rule of the hybrid mode is as follows: A buffer threshold is set for the remaining space of the buffer of the processing node. When shunting the real-time data stream, the real-time data stream is first distributed to the first processing node in the node priority sequence until the remaining space of the buffer of this processing node reaches the buffer threshold. Then, the real-time data stream is distributed to the second processing node in the node priority sequence until the remaining space of the buffer of this processing node reaches the buffer threshold. Then, the real-time data stream is distributed to the third processing node in the node priority sequence, and so on. By analogy, until the remaining space of the buffer of the last processing node in the node priority sequence reaches the buffer threshold, the real-time data stream is redistributed to the first processing node in the node priority sequence, and so on, in a cycle.

[0019] Further, the obtaining processes of the processing mean CJ and the processing floating value CF include: The first test cycle T1, the fourth test cycle T4, …, the (3m + 1)-th test cycle T3m+1 all adopt the priority mode to shunt the real-time data stream; The second test cycle T2, the fifth test cycle T5, …, the (3m + 2)-th test cycle T3m+2 all adopt the balanced mode to shunt the real-time data stream; The third test cycle T3, the sixth test cycle T6, …, the (3m + 3)-th test cycle T3m+3 all adopt the hybrid mode to shunt the real-time data stream; At the end moment of the test cycle, obtain the data processing amounts in different test cycles T3m+1 in the priority mode, sum up the data processing amounts and calculate the average value to obtain the processing mean CJ in the priority mode, and calculate the variance of the data processing amounts to obtain the processing floating value CF in the priority mode; Similarly, obtain the data processing amounts in the balanced mode and the hybrid mode and perform numerical calculations to obtain the corresponding processing mean CJ and processing floating value CF.

[0020] Further, the selection and determination process of the processing mode includes: If the management personnel adopt the efficiency-first mode to statistically process the real-time data stream, then adopt the shunt mode with the largest processing mean CJ; If the management personnel adopt the stability-first mode to statistically process the real-time data stream, then adopt the shunt mode with the smallest processing floating value CF.

[0021] Further, when performing statistical processing on the real-time data stream, a monitoring period with a fixed duration of L2 seconds is generated, and the total data input SZ within the monitoring period is obtained. A rectangular coordinate system is established with the total data input SZ as the Y-axis of the coordinate system and the monitoring period as the X-axis of the coordinate system, and a line graph of the total data input SZ - monitoring period is plotted. The slopes of the lines in the line graph of the total data input SZ - monitoring period are compared: Let the slope of any one line be G1, and the slope of the previous line be G0. The slope change rate GB is obtained through the formula GB = (G1 - G0) / G0. The slope change rate GB is compared with a preset slope change threshold GBmax: If the slope change rate GB is less than the slope change threshold GBmax, no processing is required; if the slope change rate GB is greater than or equal to the slope change threshold GBmax, it is determined that there is an increment in the real-time data stream, a traffic increase signal is generated and sent to the mobile terminal of the management personnel, and the management personnel take corresponding measures for dynamic adjustment. The adjustment measures include shortening the diversion period of the real-time data stream, changing the allocation mode, adjusting the buffer threshold in the hybrid mode, and increasing the number of processing nodes.

[0022] An efficient statistical processing method for real-time data streams, comprising the following steps:

[0023] Step 1: Mark the node that receives the real-time data stream and performs statistical processing as a processing node, perform numerical calculations on the processing speed CS, processing delay CY, and maximum throughput TT to obtain the processing coefficient CL of the processing node;

[0024] Step 2: When using the processing node to process the real-time data stream, divert the real-time data stream to different processing nodes; the diversion rules are divided into three modes: priority mode, balanced mode, and hybrid mode;

[0025] Step 3: Generate a number of test periods Tn with a fixed duration of T seconds, obtain the processing mean CJ and processing floating value CF of different test periods under different modes, and determine the processing mode according to the processing mean CJ and processing floating value CF. The processing mode includes the efficiency priority mode and the stability priority mode;

[0026] Step 4: Perform statistical processing on the real-time data stream under different processing modes, generate a monitoring period with a fixed duration of L2 seconds, plot a line graph of the total data input SZ - monitoring period, and determine whether there is an increment in the real-time data stream, generate corresponding signals, and take corresponding measures for adjustment.

[0027] The present invention has the following beneficial effects:

[0028] 1. By the processing node analysis module, comprehensively considering factors such as processing speed, processing delay, and maximum throughput, accurately evaluate the actual processing capacity of the processing node, provide a scientific basis for subsequent data diversion, and improve resource utilization efficiency;

[0029] 2. The data shunting module provides various shunting methods such as priority mode, balanced mode, and hybrid mode, which can flexibly select the most suitable shunting strategy according to different service requirements and data characteristics to achieve the balance between data processing efficiency and system stability;

[0030] 3. The shunting mode test module conducts system tests and analyzes different shunting modes, and can provide clear basis for mode selection for management personnel in both the efficiency - first and stability - first modes, enabling the system to achieve the best processing effect in different scenarios;

[0031] 4. The dynamic monitoring and adjustment module dynamically monitors the real - time data stream. When the total amount of data input changes significantly, it can send signals to management personnel in a timely manner and take corresponding measures to ensure that the system always adapts to the change of data flow and guarantee the timeliness and accuracy of processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0033] Figure 1 It is the system block diagram of Embodiment 1 of the present invention;

[0034] Figure 2 It is the method flow chart of Embodiment 2 of the present invention;

[0035] Figure 3 It is the line chart of the total amount of data input SZ - monitoring period in Embodiment 1 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0037] Embodiment 1: As Figure 1 shown, an efficient statistical processing system for real - time data stream includes a statistical analysis platform, and the statistical analysis platform is communicatively connected with a processing node analysis module, a data shunting module, a shunting mode test module, and a dynamic monitoring and adjustment module;

[0038] The processing node analysis module is used to analyze and evaluate the capabilities of the processing nodes for real-time data streams: Nodes that receive real-time data streams and perform statistical processing are marked as processing nodes, and the processing speed CS, processing latency CY, and maximum throughput TT of the processing nodes are obtained; The processing speed CS represents the average number of bytes of data processed by the processing node within a preset time; The processing latency CY represents the average time delay from when the data enters the processing node to when the processing is completed and the result is output; The maximum throughput TT represents the maximum data traffic that the processing node can handle;

[0039] Obtain the historical monitoring data of the processing nodes, read the historical monitoring data from a CSV file, which includes fields: including processing speed CS, processing latency CY, maximum throughput TT, and processing coefficient CL;

[0040] Take 70% of the sample data in the historical monitoring data as the training set and 30% of the sample data as the validation set;

[0041] Construct a multi-layer perceptron (MLP) model, input the sample data in the training set into the model for model training, and optimize the parameters of the model by minimizing the loss function;

[0042] Among them, the multi-layer perceptron (MLP) model includes an input layer (processing speed CS, processing latency CY, and maximum throughput TT), a hidden layer, and an output layer (processing coefficient CL);

[0043] Then use the validation set to validate the trained multi-layer perceptron neural network model;

[0044] Use the trained model to predict the processing speed CS, processing latency CY, and maximum throughput TT of the current processing node to obtain the processing coefficient CL;

[0045] Among them, taking the processing speed CS, processing latency CY, and maximum throughput TT of each processing node as the source data, construct the sample data (CSi, CYi, TTi) of each processing node, where CSi represents the processing speed of the i-th processing node in the historical data, CYi represents the processing latency of the i-th processing node in the historical data, and TTi represents the maximum throughput of the i-th processing node in the historical data.

[0046] Through the processing node analysis module, comprehensively considering factors such as processing speed, processing latency, and maximum throughput, accurately evaluate the actual processing capabilities of the processing nodes, provide a scientific basis for subsequent data shunting, and improve resource utilization efficiency.

[0047] The data shunting module is used to perform shunting processing on real-time data streams in different modes: when using processing nodes to process real-time data streams, the real-time data streams are shunted to different processing nodes; the shunting rules are based on the processing coefficient CL of the processing nodes and are divided into three modes: priority mode, balanced mode, and hybrid mode;

[0048] Specifically, the shunting rule for the priority mode is as follows: the area used to temporarily store data in the processing node is marked as a buffer, the processing nodes are arranged in descending order of the processing coefficient CL to obtain a node priority sequence, and when shunting the real-time data stream, it is preferentially distributed to the processing node with a smaller serial number in the node priority sequence and with remaining storage space in the buffer;

[0049] The shunting rule for the balanced mode is: when shunting the real-time data stream, a shunting period with a fixed duration of L1 seconds is generated. The real-time data stream in the first shunting period is distributed to the first processing node in the node priority sequence, the real-time data stream in the second shunting period is distributed to the second processing node in the node priority sequence, and so on, until after the last processing node is allocated the real-time data stream, the real-time data stream in the next shunting period is distributed to the first processing node in the node priority sequence, and so on, in a cycle;

[0050] The shunting rule for the hybrid mode is: a buffer threshold is set for the remaining space of the buffer of the processing node. When shunting the real-time data stream, the real-time data stream is first distributed to the first processing node in the node priority sequence until the remaining space of the buffer of this processing node reaches the buffer threshold, then the real-time data stream is distributed to the second processing node in the node priority sequence until the remaining space of the buffer of this processing node reaches the buffer threshold, then the real-time data stream is distributed to the third processing node in the node priority sequence, and so on, until the remaining space of the buffer of the last processing node in the node priority sequence reaches the buffer threshold, then the real-time data stream is redistributed to the first processing node in the node priority sequence, and so on, in a cycle; Through the data shunting module, multiple shunting methods such as priority mode, balanced mode, and hybrid mode are provided, which can flexibly select the most suitable shunting strategy according to different business requirements and data characteristics, and achieve the balance of data processing efficiency and system stability.

[0051] The shunt mode test module is used to test and analyze different shunt modes: before the real-time data stream is statistically processed by the processing node, several test cycles Tn with a fixed duration of T seconds are generated, where n represents the number of test cycles, n = 1, 2, ……, k, and k is a positive integer; the first test cycle T1, the fourth test cycle T4, ……, the (3m + 1)-th test cycle T3m+1 all adopt the priority mode for shunting the real-time data stream; the second test cycle T2, the fifth test cycle T5, ……, the (3m + 2)-th test cycle T3m+2 all adopt the balanced mode for shunting the real-time data stream; the third test cycle T3, the sixth test cycle T6, ……, the (3m + 3)-th test cycle T3m+3 all adopt the hybrid mode for shunting the real-time data stream;

[0052] At the end of the test cycle, the data processing volumes in different test cycles T3m+1 under the priority mode are obtained, and the sum of the data processing volumes is averaged to calculate the processing mean CJ under the priority mode, and the variance of the data processing volumes is calculated to obtain the processing floating value CF under the priority mode; similarly, the data processing volumes under the balanced mode and the hybrid mode are obtained and numerically calculated to obtain the corresponding processing mean CJ and processing floating value CF; when the processing node statistically processes the real-time data stream, the processing modes include the efficiency-first mode and the stability-first mode; if the management personnel adopt the efficiency-first mode to statistically process the real-time data stream, the shunt mode with the largest processing mean CJ is adopted; if the management personnel adopt the stability-first mode to statistically process the real-time data stream, the shunt mode with the smallest processing floating value CF is adopted; through the system test and analysis of different shunt modes by the shunt mode test module, it is possible to provide a clear basis for mode selection for the management personnel under both the efficiency-first and stability-first modes, so that the system can achieve the best processing effect in different scenarios.

[0053] Such as Figure 3As shown in the figure, the dynamic monitoring and adjustment module is used to dynamically adjust the statistical processing according to the monitoring data of the real-time data stream: when performing statistical processing on the real-time data stream, a monitoring period with a fixed duration of L2 seconds is generated, and the total data input SZ within the monitoring period is obtained. A rectangular coordinate system is established with the total data input SZ as the Y-axis of the coordinate system and the monitoring period as the X-axis of the coordinate system, and a line graph of total data input SZ - monitoring period is plotted; the slopes of the lines in the line graph of total data input SZ - monitoring period are compared: the slope of any one line is set as G1, and the slope of the previous line is set as G0. The slope change rate GB is obtained through the formula GB = (G1 - G0) / G0; the slope change rate GB is compared with the preset slope change threshold GBmax: if the slope change rate GB is less than the slope change threshold GBmax, no processing is required; if the slope change rate GB is greater than or equal to the slope change threshold GBmax, it is determined that there is an increment in the real-time data stream, a traffic increase signal is generated and sent to the mobile terminal of the management personnel, and the management personnel take corresponding measures for dynamic adjustment. The adjustment measures include shortening the diversion period of the real-time data stream, changing the distribution mode, adjusting the buffer threshold in the hybrid mode, and increasing the number of processing nodes; through the dynamic monitoring and adjustment module, the real-time data stream is dynamically monitored. When the total data input changes significantly, a signal can be sent to the management personnel in a timely manner and corresponding measures are taken to ensure that the system always adapts to the change of data traffic and guarantee the timeliness and accuracy of processing.

[0054] Embodiment 2: As Figure 2 shown, an efficient statistical processing method for real-time data stream includes the following steps:

[0055] Step 1: Mark the node that receives the real-time data stream and performs statistical processing as the processing node, perform numerical calculations on the processing speed CS, processing delay CY, and maximum throughput TT to obtain the processing coefficient CL of the processing node;

[0056] Step 2: When using the processing node to process the real-time data stream, divert the real-time data stream to different processing nodes; the diversion rules are divided into three modes: priority mode, balanced mode, and hybrid mode;

[0057] Step 3: Generate a number of test periods Tn with a fixed duration of T seconds, obtain the processing mean CJ and processing floating value CF of different test periods under different modes, and determine the processing mode according to the processing mean CJ and processing floating value CF. The processing mode includes the efficiency priority mode and the stability priority mode;

[0058] Step 4: Generate a monitoring period with a fixed duration of L2 seconds, plot a line graph of total data input SZ - monitoring period and determine whether there is an increment in the real-time data stream, generate a corresponding signal and take corresponding measures for adjustment.

[0059] An efficient statistical processing system for real-time data streams. When working, the nodes that receive real-time data streams and perform statistical processing are marked as processing nodes. The processing coefficient CL of the processing nodes is obtained, and the real-time data streams are split into different processing nodes according to the processing coefficient CL. The splitting rules include a priority mode, a balanced mode, and a hybrid mode. A number of test cycles Tn with a fixed duration of T seconds are generated. The processing mean CJ and the processing floating value CF of different test cycles in different modes are obtained, and the processing mode is determined according to the processing mean CJ and the processing floating value CF. The processing modes include an efficiency-priority mode and a stability-priority mode. A monitoring cycle with a fixed duration of L2 seconds is generated. A line chart of the total data input SZ - monitoring cycle is plotted and it is judged whether there is an increment in the real-time data stream, and corresponding signals are generated and corresponding measures are taken for adjustment.

[0060] The above content is only an example and explanation of the structure of the present invention. Those skilled in the art of this technology can make various modifications or supplements to the described specific embodiments or use similar methods for substitution, as long as they do not deviate from the structure of the invention or exceed the scope defined by this claim book, they should all fall within the protection scope of the present invention.

[0061] In the description of this specification, the description with reference to terms such as "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0062] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not elaborate on all details, nor do they limit the invention to only the specific implementation manners. Obviously, according to the content of this specification, many modifications and changes can be made. This specification selects and specifically describes these embodiments in order to better explain the principle and practical application of the present invention, so that those skilled in the art of this technology can well understand and utilize the present invention. The present invention is only limited by the claim book and its full scope and equivalents.

Claims

1. An efficient statistical processing system for real-time data streams, characterized in that, Including: A processing node analysis module for analyzing and evaluating the capabilities of processing nodes in real-time data streams: Nodes that receive and statistically process real-time data streams are marked as processing nodes, and the processing coefficient CL of the processing nodes is obtained; A data shunting module for shunting real-time data streams in different modes: When using processing nodes to process real-time data streams, the real-time data streams are shunted to different processing nodes; The shunting rules are divided into a priority mode, a balanced mode, and a hybrid mode based on the processing coefficient CL of the processing nodes; A shunting mode testing module for testing and analyzing different shunting modes: Generating a number of test cycles Tn with a fixed duration of T seconds; Obtaining the processing mean CJ and the processing floating value CF of different test cycles under different shunting modes, and determining the processing mode based on the processing mean CJ and the processing floating value CF; The process of obtaining the processing mean CJ and the processing floating value CF includes: The first test cycle T1, the fourth test cycle T4,..., the (3m + 1)-th test cycle T3m+1 all use the priority mode to shunt the real-time data stream; The second test cycle T2, the fifth test cycle T5,..., the (3m + 2)-th test cycle T3m+2 all use the balanced mode to shunt the real-time data stream; The third test cycle T3, the sixth test cycle T6,..., the (3m + 3)-th test cycle T3m+3 all use the hybrid mode to shunt the real-time data stream; At the end of the test cycle, obtain the data processing amounts of different test cycles T3m+1 in the priority mode, and calculate the average value by summing the data processing amounts to obtain the processing mean CJ in the priority mode; Calculating the variance of the data processing amounts to obtain the processing floating value CF in the priority mode; Obtaining the data processing amounts in the balanced mode and the hybrid mode and performing numerical calculations to obtain the corresponding processing mean CJ and processing floating value CF; The processing modes include an efficiency priority mode and a stability priority mode; Performing statistical processing on the real-time data stream in different processing modes, generating a monitoring cycle with a fixed duration of L2 seconds, plotting a data input total amount SZ - monitoring cycle line chart and judging whether there is an increment in the real-time data stream, generating corresponding signals and taking corresponding measures for adjustment.

2. The efficient statistical processing system for real-time data streams according to claim 1, characterized in that, The process of obtaining the processing coefficient CL of the processing nodes includes: Obtaining the processing speed CS, the processing delay CY, and the maximum throughput TT of the processing nodes, and inputting the processing speed CS, the processing delay CY, and the maximum throughput TT into a multi-layer perceptron model to obtain the processing coefficient CL of the processing nodes.

3. An efficient statistical processing system for real-time data streams according to claim 2, characterized in that, The processing speed CS represents the average number of bytes of data processed by the processing node within a preset time; The processing delay CY represents the average time delay from when the data enters the processing node to when the processing is completed and the result is output; The maximum throughput TT represents the maximum data flow that the processing node can process.

4. An efficient statistical processing system for real-time data streams according to claim 1, characterized in that, The shunting rule of the priority mode is specifically: Marking the area for temporarily storing data in the processing node as a buffer, and arranging the processing nodes in descending order of the processing coefficient CL to obtain a node priority sequence; When shunting real-time data streams, give priority to the processing nodes with smaller serial numbers in the node priority sequence and with remaining storage space in the buffer.

5. An efficient statistical processing system for real-time data streams according to claim 1, characterized in that, The shunting rule for the balanced mode is: When shunting real-time data streams, generate a shunting period with a fixed duration of L1 seconds. The real-time data stream in the first shunting period is assigned to the first processing node in the node priority sequence, the real-time data stream in the second shunting period is assigned to the second processing node in the node priority sequence, and so on. By analogy, until the last processing node is assigned the real-time data stream, the real-time data stream in the next shunting period is assigned to the first processing node in the node priority sequence, and so on, in a cycle.

6. An efficient statistical processing system for real-time data streams according to claim 1, characterized in that, The shunting rule for the hybrid mode is: Set a buffer threshold for the remaining space in the buffer of the processing node. When shunting real-time data streams, first assign the real-time data stream to the first processing node in the node priority sequence. When the remaining space in the buffer of this processing node reaches the buffer threshold, assign the real-time data stream to the second processing node in the node priority sequence. When the remaining space in the buffer of this processing node reaches the buffer threshold, assign the real-time data stream to the third processing node in the node priority sequence, and so on. By analogy, until the remaining space in the buffer of the last processing node in the node priority sequence reaches the buffer threshold, re-assign the real-time data stream to the first processing node in the node priority sequence, in a cycle.

7. An efficient statistical processing system for real-time data streams according to claim 1, characterized in that, The process of selecting and determining the processing mode includes: If the management personnel adopt the efficiency-first mode to statistically process the real-time data stream, then adopt the shunting mode with the largest processing mean CJ. If the management personnel adopt the stability-first mode to statistically process the real-time data stream, then adopt the shunting mode with the smallest processing floating value CF.

8. An efficient statistical processing system for real-time data streams according to claim 7, characterized in that, When statistically processing the real-time data stream, generate a monitoring period with a fixed duration of L2 seconds, obtain the total data input SZ within the monitoring period, establish a rectangular coordinate system with the total data input SZ as the Y-axis of the coordinate system and the monitoring period as the X-axis of the coordinate system, and draw a total data input SZ - monitoring period line graph. Compare the slopes of the lines in the total data input SZ - monitoring period line graph: Set the slope of any line as G1, and the slope of the previous line as G0. Obtain the slope change rate GB through the formula GB = (G1 - G0) / G0; compare the slope change rate GB with the preset slope change threshold GBmax: If the slope change rate GB is less than the slope change threshold GBmax, no processing is required; If the slope change rate GB is greater than or equal to the slope change threshold GBmax, then it is determined that there is an increment in the real-time data stream, generate a traffic increase signal and send the signal to the mobile terminal of the management personnel, and the management personnel take corresponding measures for dynamic adjustment. The adjustment measures include shortening the shunting period of the real-time data stream, changing the allocation mode, adjusting the buffer threshold in the hybrid mode, and increasing the number of processing nodes.

9. An efficient statistical processing method for real-time data streams, characterized in that, Include the following steps: Step 1: Mark the node that receives the real-time data stream and performs statistical processing as the processing node, calculate the numerical values of the processing speed CS, processing delay CY, and maximum throughput TT, and obtain the processing coefficient CL of the processing node. Step 2: When using the processing node to process the real-time data stream, split the real-time data stream into different processing nodes; the splitting rules are divided into three modes: priority mode, balanced mode, and hybrid mode. Step 3: Generate a number of test cycles Tn with a fixed duration of T seconds, obtain the processing mean CJ and processing floating value CF of different test cycles under different modes, and determine the processing mode according to the processing mean CJ and processing floating value CF. The process of obtaining the processing mean CJ and processing floating value CF includes: The first test cycle T1, the fourth test cycle T4,..., the (3m + 1)-th test cycle T3m+1 all use the priority mode to split the real-time data stream; the second test cycle T2, the fifth test cycle T5,..., the (3m + 2)-th test cycle T3m+2 all use the balanced mode to split the real-time data stream; the third test cycle T3, the sixth test cycle T6,..., the (3m + 3)-th test cycle T3m+3 all use the hybrid mode to split the real-time data stream; at the end of the test cycle, obtain the data processing volume of different test cycles T3m+1 under the priority mode, and calculate the average value of the sum of the data processing volumes to obtain the processing mean CJ under the priority mode. Calculate the variance of the data processing volume to obtain the processing floating value CF under the priority mode. Obtain the data processing volume under the balanced mode and hybrid mode and perform numerical calculations to obtain the corresponding processing mean CJ and processing floating value CF. The processing modes include the efficiency priority mode and the stability priority mode. Step 4: Perform statistical processing on the real-time data stream under different processing modes, generate a monitoring cycle with a fixed duration of L2 seconds, draw a line chart of the total data input SZ - monitoring cycle, and determine whether there is an increment in the real-time data stream, generate corresponding signals and take corresponding measures for adjustment.

Images