Data processing system for processing mass data and processing method thereof

Abstract

The invention discloses a data processing system for processing mass data and a processing method thereof, and relates to the technical field of data processing, the system comprises the following components: a data receiving module for monitoring a data access port, receiving data from an external data source, and performing primary processing on the received data for subsequent processing, according to the invention, the intelligent data routing module utilizes a reinforcement learning model to make an optimal routing decision based on data characteristics and real-time load conditions so as to ensure that data can be efficiently processed by specified nodes in a processed node cluster, and the load balancing module monitors and processes the load conditions of the node cluster in real time, so that the load balancing efficiency of the node cluster is improved. According to the method and the system, the future load peak is predicted, and the data routing strategy is intelligently adjusted according to the analysis result, so that the performance bottleneck and the fault risk caused by overload are effectively avoided, and the intelligent data routing and load balancing mechanism enables the system to keep efficient performance and stable operation when processing a large amount of data.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing system and a processing method thereof for processing a large amount of data. Background Art

[0002] Traditional data reception methods often lack strict data integrity and authentication mechanisms and are easily interfered by malicious data or outliers, which reduces the accuracy and reliability of data processing. They also lack in-depth analysis and understanding of data characteristics and are unable to fully utilize the inherent properties and characteristics of the data to optimize the data processing process, thus affecting the efficiency and performance of data processing.

[0003] Traditional data processing usually relies on fixed routing rules and load balancing strategies, which cannot be flexibly adjusted according to the real-time characteristics of data and the load conditions of processing nodes. When processing large amounts of data, this fixed routing and load balancing mechanism often causes some processing nodes to be overloaded while other nodes are idle, thereby reducing overall performance and stability.

[0004] Therefore, in view of the deficiencies and shortcomings of the prior art, the present invention proposes a data processing system and a processing method for processing large amounts of data to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to make up for the shortcomings of the prior art and provide a data processing system and a processing method for processing large amounts of data. The system can receive and preprocess data through a monitoring port, use a feature analysis module to identify data types and sources, and an intelligent routing module combines real-time load and reinforcement learning models to optimize data routing to ensure efficient data processing. The load balancing module intelligently adjusts strategies to achieve efficient use of system resources.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: On the one hand, a data processing system for processing a large amount of data includes the following components:

[0007] Data receiving module: monitors the data access port, receives data from external data sources, performs preliminary processing on the received data, and stores the processed data for subsequent processing;

[0008] Data characteristic analysis module: determines the data size through byte counting, identifies the data type, analyzes the source and other identifications of the data, and transmits the analysis results to the intelligent data routing module;

[0009] Intelligent Data Routing Module: Receives the characteristic information of the data from the Data Characteristic Analysis Module, obtains the real-time load condition of the current system from the Load Balancing Module, makes routing decisions using a reinforcement learning model, sends the data to the specified processing node according to the routing decision, and obtains the node status information from the processing node cluster in real time to ensure that the data can be processed efficiently;

[0010] Processing Node Cluster: Receives data processing instructions and data from the Intelligent Data Routing Module, executes corresponding data processing tasks according to the instructions, and outputs the processing results to the specified location;

[0011] Load Balancing Module: Analyzes and calculates the load information of the processing node cluster, evaluates the load condition of the current system, predicts future load peaks based on historical data, and intelligently adjusts the data routing strategy according to the results of real-time load analysis and predictive analysis to achieve load balancing.

[0012] Further, the Data Receiving Module uses a network programming library to listen on a specified port, receives external data sources in the form of byte streams, verifies the integrity of the data, checks the specific identifiers of the data to ensure that the data comes from the expected data source, uses the IQR rule to identify and process outliers, divides the data set into four equal numerical points, Q1 is the value at 25% of the data set, Q3 is the value at 75% of the data set, IQR is the difference between Q3 and Q1, and its mathematical formula is: IQR = Q3 - Q1. The screening condition for outliers based on the IQR rule: x i < Q1 - 1.5×IQR or x i > Q1 - 1.5×IQR values, where: x i is each data point in the data set, replaces the identified outliers, normalizes the data, and stores the processed data in the database.

[0013] Even further, the Data Characteristic Analysis Module determines the size of the data by calculating the number of bytes of the data, uses a content parsing algorithm to identify the data type, extracts the unique identifier according to the format and content of the data, calculates the arrival rate or traffic of the data using exponential smoothing statistical techniques to identify network congestion and abnormal behaviors, and transmits the analyzed data to the Intelligent Data Routing Module according to the characteristics of size, type, source IP address, and device ID.

[0014] Even further, the Data Characteristic Analysis Module calculates the arrival rate and traffic of the data using exponential smoothing statistical techniques, determines the smoothing coefficient α according to the actual situation, and α determines the weights of new data and old data in the prediction. Let So be the initial smoothing value, that is, the actual arrival rate in the first period. For each time period t (t = 1, 2, 3,...), the smoothing value calculation formula is: S n = α × y n+(1-α)×S n-1 , where: S n is the smoothed value of period t, y n is the actual arrival rate in period t·S n-1 It is the smoothed value of the t-1th period. The traffic pattern is analyzed by continuously calculating the smoothed values ​​of multiple time periods. A stable smoothed value indicates normal network traffic. A sudden change in the smoothed value indicates network congestion and abnormal behavior. According to historical data and network specifications, normal and abnormal thresholds of the arrival rate are set. If the calculated smoothed value exceeds or falls below these thresholds, an alarm is triggered.

[0015] Furthermore, the intelligent data routing module receives characteristic information of the data, parses the received characteristic information to understand the nature and processing requirements of the data, obtains the load status, processing capacity, and available resources of the current system from the load balancing module, evaluates the load status of the current system based on the collected load information, and chooses to use the Q-learning model to make routing decisions. Based on the routing decision, the data is sent to the designated processing node, and the node status information is obtained from the processing node cluster in real time. Based on the collected node status information, the health status of the processing node is evaluated, and the node status label is updated. If an abnormality is detected in a node, corresponding measures will be taken.

[0016] Furthermore, the intelligent data routing module uses a Q-learning model to make routing decisions, defines a state set according to the characteristics of the network and the processing nodes, determines the action set taken in each state, sends data to a specific processing node, selects a certain routing strategy, initializes a Q value table for each state-action, and sets the Q value to a small random value. It receives characteristic information about the data to be processed from the data characteristic analysis module, obtains the real-time load of the current system from the load balancing module, determines the current state based on the received data characteristics and the real-time load, uses the ε-greedy strategy to select an action, randomly selects an action with a probability of ε, selects the action with the maximum Q value in the current state with a probability of 1-ε, routes the data to the corresponding processing node according to the selected action, calculates the reward according to the result after the routing decision, observes and determines the next state after the action is executed, and updates the Q value table using the Q value update formula, which is: Where: Q(s,a) is the Q value of the current state s and action a, α is the learning rate, which determines the degree of coverage of the old Q value by the new information, r is the immediate reward obtained from the environment after executing action a, γ is the discount factor, which is used to balance the importance of immediate rewards and future rewards, A is the set of all actions, s′ is the new state reached after executing action a, It represents the maximum Q value of all actions in the new state s′. Repeat the above steps. Through continuous trial and error and learning, the intelligent data routing module will gradually find the optimal routing strategy.

[0017] Furthermore, the processing node cluster listens to data processing instructions and data from the intelligent data routing module through a network interface, parses the data, verifies the integrity, security and format correctness of the data, extracts the data to be processed from the data after verification, uses a load balancing algorithm to allocate data processing tasks, processes the data according to the task type application, formats the processing results into a format understood by the intelligent data routing module, outputs them to a specified location, sends its own status information back to the intelligent data routing module in real time, and when a processing node fails, the intelligent data routing module triggers a fault recovery mechanism to reallocate the task to other healthy nodes.

[0018] Furthermore, the load balancing module processes the node load information in real time through a specific API interface and performs standardization on the collected load information so that indicators of different dimensions can be compared. The standardization formula is: Where: l′ ij is the normalized load value, l ij is the original load value, μ j is the mean of the jth index, σ j is the standard deviation of the jth indicator. According to the importance of the indicator and business requirements, a weight is assigned to each indicator, and the weighted sum is used to obtain the comprehensive load value. According to the comprehensive load value of each node, the comprehensive load value calculation formula is: Where: Load i is the comprehensive load value of node i, w j It is the weight of the jth indicator. The average load and load variance are calculated by comprehensive load values, and the load status of the system is analyzed. The ARIMA model is constructed using historical load data to predict the load trend in the future. According to the results of real-time load analysis and predictive analysis, the data routing strategy is adjusted and sent to the intelligent data routing module. The intelligent data routing module performs data routing according to these guidance information to ensure that tasks can be assigned to nodes with lower loads to achieve load balancing.

[0019] Furthermore, the load balancing module generates a set {h t}, where: t = 1, 2, ..., T, check the stationarity of the data. When the data is not stationary, perform a difference operation until the data becomes stationary. Determine the values ​​of p and q by looking at the graphs of the autocorrelation function ACF and partial autocorrelation function PACF of the differential set. Use historical data to estimate the parameters of the ARIMA model. The ARIMA (p, d, q) model is expressed as: φ (B) (1-B) d h t =θ(B)∈ t , where: φ(B) = 1-φ1B-φ2B 2 -…-φ p B p is an autoregressive polynomial, θ(B) = 1-θ1B-θ2B 2 -…-θ q B q is the moving average polynomial, B is the backshift operator, ∈ t is the error term, which is usually assumed to be a white noise process. Check whether the residual of the model satisfies the white noise assumption. When the model is verified, use the estimated model parameters to predict future load values. T+1 The prediction is expressed as: in: Yes T+1 The predicted value of T+1-j is the error term.

[0020] On the other hand, a data processing method for processing a large amount of data, the specific steps of the processing method are:

[0021] Data reception and verification: Use advanced network programming libraries to listen to designated ports and receive data streams in the form of byte streams. During the reception process, data integrity verification and specific identifier checks are performed to ensure that the received data originates from the expected data source, and outliers are identified and processed, thereby enhancing data security.

[0022] Data characteristic analysis: Use byte counting, content analysis algorithm and exponential smoothing statistical technology to conduct a comprehensive and in-depth characteristic analysis of the received data. Through this step, the inherent characteristics of the data can be accurately understood, providing rich decision-making basis for subsequent intelligent data routing;

[0023] Intelligent data routing: Based on the reinforcement learning model, it can analyze data characteristics and the load of the current processing node cluster in real time, dynamically generate the optimal data routing strategy, and ensure that data can be processed efficiently and accurately by designated nodes by intelligently selecting processing nodes, thereby improving the overall processing efficiency of the system;

[0024] Load balancing optimization: Real-time monitoring of the load status of the processing node cluster, using prediction algorithms to predict future load peaks, intelligently adjusting data routing strategies based on prediction results and real-time data analysis, and achieving dynamic optimization of load balancing, avoiding performance bottlenecks and failure risks caused by node overload, and ensuring stable operation under high load conditions

[0025] Compared with the prior art, the data processing system and the processing method for processing a large amount of data have the following beneficial effects:

[0026] 1. The present invention uses a network programming library to listen to a designated port, receives data streams in the form of byte streams, and uses algorithms to verify the integrity and specific identifiers of the data, thereby ensuring that the received data comes from the expected data source and effectively identifying and processing outliers. The data characteristic analysis module uses byte counting, content parsing algorithm, and exponential smoothing statistical technology to perform a comprehensive characteristic analysis of the data, providing a rich decision-making basis for the intelligent data routing module, which not only improves the accuracy of data processing, but also enhances the reliability and security of the system.

[0027] 2. The present invention utilizes a reinforcement learning model through an intelligent data routing module to make optimal routing decisions based on data characteristics and real-time load conditions, ensuring that data can be efficiently processed by designated nodes in the processing node cluster. The load balancing module monitors and processes the load conditions of the node cluster in real time, predicts future load peaks, and intelligently adjusts the data routing strategy based on the analysis results, thereby effectively avoiding performance bottlenecks and failure risks caused by overload. This intelligent data routing and load balancing mechanism enables the system to maintain efficient performance and stable operation when processing large amounts of data.

[0028] Other advantages, objectives and features of the present invention will be set forth in part in the following description and, in part, will be apparent to those skilled in the art based on an examination of the following or may be taught from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 A flow chart of a data processing system for processing large amounts of data.

[0031] Figure 2A flowchart of a data processing method for processing a large amount of data. Detailed implementation mode

[0032] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0033] Embodiment 1:

[0034] This embodiment provides a data processing system for processing a large amount of data. The data receiving module receives external data through a listening port and performs preliminary processing. The data characteristic analysis module identifies information such as the data type and source, providing a decision basis for intelligent routing. The intelligent data routing module, based on a reinforcement learning model and combined with the real-time load status, intelligently selects processing nodes to ensure efficient data processing. The processing node cluster is responsible for executing data processing tasks, and the load balancing module analyzes the load information of the processing node cluster, predicts and adjusts the data routing strategy to achieve system load balancing as Figure 1 .

[0035] Data receiving module: The used network programming library listens to a specified TCP port. The external data source sends data in the form of a byte stream through this port. After the module receives the data stream, it verifies the integrity of the data to ensure that the data has not been tampered with during transmission. The module checks specific identifiers in the data, such as the data source ID, to ensure that the data comes from the expected data source. The module uses the IQR (Interquartile Range) rule to detect outliers in the received data. The data set is divided into four equal parts of numerical points. Q1 is the value at the 25% position in the data set, Q3 is the value at the 75% position in the data set, and IQR is the difference between Q3 and Q1. Its mathematical formula is: IQR = Q3 - Q1. The screening condition for outliers based on the IQR rule: x i < Q1 - 1.5×IQR or x i > Q1 - 1.5×IQR values. Replace the abnormal data to ensure data quality, and store the processed data in the database for subsequent processing.

[0036] Data characteristic analysis module: read the data to be processed from the database, determine the size of the data by calculating the number of bytes of the data, use the content parsing algorithm to identify the data type, extract the unique identifier such as the source IP address and device ID according to the format and content of the data, and use the exponential smoothing statistical technology to calculate the arrival rate or flow of the data. The smoothing coefficient α determines the weight of new data and old data in the prediction. Let S0 be the actual arrival rate. For each time period t (t=1, 2, 3, ...), the smoothing value formula is: S n =α×y n +(1-α)×S n-1 ,The traffic pattern is analyzed by continuously calculating the smoothed values ​​of multiple time periods. ,If the smoothed value is stable, it indicates that the network traffic is normal. ,If the smoothed value changes suddenly, it indicates that there is ,network congestion or abnormal behavior. ,The data characteristic information obtained from the analysis is transmitted to the ,intelligent data routing module.

[0037] Intelligent data routing module: Receives data characteristic information from the data characteristic analysis module, obtains the current system load status, processing capacity and available resource information from the load balancing module, and uses the Q-learning model to make routing decisions based on this information. Define a set of states, determine the set of actions that can be taken to send data to a specific processing node in each state, initialize a Q value table for each state-action pair, and set it to a small random value. At runtime, determine the current state based on the received data characteristics and real-time load conditions, use the ε-greedy strategy to select an action, route the data to the specified processing node based on the selected action, and obtain node status information from the processing node cluster in real time to evaluate the health of the processing node and update the node status label.

[0038] Processing node cluster: It is composed of multiple high-performance computing nodes and is responsible for executing specific data processing tasks. When the intelligent data routing module sends data to a processing node, the node performs the corresponding processing tasks according to the received data processing instructions and data, and outputs the processing results to the specified location. When a processing node fails or an error occurs, the intelligent data routing module triggers the fault recovery mechanism and reallocates the task to other healthy nodes.

[0039] Load balancing module: Through a specific API interface, it processes node load information in real time and standardizes the collected load information so that indicators of different dimensions can be compared. The standardization formula is: Assign a weight to each indicator and use weighted sum to get the comprehensive load value. The calculation formula for the comprehensive load value is: By combining the load value calculation and load variance, the average load mathematical expression is: A high average load indicates that the nodes are processing a large number of tasks and the overall resource utilization is high. A low average load indicates that the resources are underutilized. Analyze the load status of the system. The mathematical expression of load variance is: The load variance indicates the degree of dispersion of the node load. A larger variance indicates that the load differences between nodes are large, and a smaller variance indicates that the load is more balanced.

[0040] Construct an ARIMA model to predict the load trend in the future and use historical load data to generate a set {h t}, where: t = 1, 2, ..., T, check the stability of the data. If the data is not stable, perform a differential operation until the data becomes stable. The differential mathematical formula is: Δh t =h t -h t-1 , the values ​​of p and q are determined by the graphs of the autocorrelation function ACF and partial autocorrelation function PACF of the differential set, and the parameters of the ARIMA model are estimated using historical data. The ARIMA (p, d, q) model is expressed as: φ(B)(1-B) d h t =θ(B)∈ t , by checking whether the residual of the model meets the white noise assumption, we can determine whether the model meets the requirements. When the model is verified, we use the estimated model parameters to predict the future load value. T+1 The prediction is expressed as: According to the results of real-time load analysis and predictive analysis, the data routing strategy is adjusted to ensure that tasks can be assigned to nodes with lower loads, thereby achieving load balancing.

[0041] Embodiment 2:

[0042] In this embodiment, a data processing system for processing a large amount of data is provided, with particular emphasis on an intelligent data routing module:

[0043] The intelligent data routing module receives data characteristic information from the data characteristic analysis module, analyzes the received characteristic information to deeply understand the nature and processing requirements of the data, obtains the real-time load status, processing capacity and available resource information of the current system from the load balancing module, and conducts a comprehensive assessment of the overall load status of the current system based on the collected load information.

[0044] We choose to use the Q-learning model to make data routing decisions. According to the characteristics of the network and processing nodes, we define the state set, which reflects the different conditions of the system and the network. We determine the set of actions that can be taken in each state. The action set refers to which specific processing node to send the data to or what routing strategy to adopt. We initialize a Q value table for each state-action pair and set it to a small random value. The Q value will be used to guide future routing decisions.

[0045] When new data arrives, the current state is determined based on the characteristics of the received data and the real-time load situation. The ε-greedy strategy is used to select actions. An action is randomly selected with a probability of ε to explore a better solution. The action with the maximum Q value in the current state is selected with a probability of 1-ε. Using existing knowledge, the data is routed to the corresponding processing node according to the selected action. After the routing decision is executed, the results are observed and recorded, especially the key performance indicators of the network's delay, throughput, and data loss rate. The reward value is calculated based on the actual performance results after the routing decision. The reward value refers to the network's delay, throughput, and data loss rate. The Q value table is updated using the Q value update formula to utilize these experiences in future routing decisions. The Q value update formula is: The Q-value update formula takes into account the current Q-value, learning rate, immediate reward, discount factor, and the maximum Q-value factor in the new state.

[0046] By repeating the above steps, through continuous trial and error and learning process, the intelligent data routing module will gradually find and optimize its routing strategy. As time goes by and experience accumulates, the system will be able to make optimal routing decisions more effectively based on data characteristics and system load conditions.

[0047] Embodiment three:

[0048] This embodiment provides a data processing method for processing a large amount of data, the method comprising the following steps: data reception and verification, data characteristic analysis, intelligent data routing, and load balancing optimization:

[0049] First, we enter the data reception and verification phase. We use an advanced network programming library to start listening to the specified port and receive the data stream incoming from the port in the form of a byte stream. During the reception process, we perform integrity verification and checks on each received data packet to ensure that the data has not been tampered with during transmission. We perform specific identifier checks to confirm whether the source of the data packet is consistent with the expected data source and identify and process outliers in the data.

[0050] Next, we enter the data characteristic analysis phase, where we conduct preliminary processing on the verified data, including byte counting to obtain basic data statistical information, apply content parsing algorithms to deeply parse the data content, extract key information, metadata, and characteristic values, and use exponential smoothing statistical techniques to predict and analyze the historical and future trends of the data to reveal the intrinsic characteristics and potential patterns of the data, and organize the analysis results into a data feature set that can be used for subsequent decision-making.

[0051] Subsequently, the intelligent data routing stage is entered, the reinforcement learning model is activated, the current data feature analysis results and the real-time load of the processing node cluster are input, the data are analyzed in real time, and the reinforcement learning algorithm is used to dynamically generate the optimal data routing strategy. According to the routing strategy, the most suitable processing node is intelligently selected to ensure that the data can be processed efficiently and accurately, the data is sent to the selected processing node, and the relevant information of the routing decision is recorded.

[0052] Finally, enter the load balancing optimization stage, monitor the load status of each node in the processing node cluster in real time, collect performance indicators, use prediction algorithms to analyze historical load data, predict future load peaks and change trends, and intelligently adjust data routing strategies based on prediction results and real-time data analysis, such as reallocating data to nodes with lower loads to balance the load of the entire cluster, achieve dynamic optimization of load balancing, and ensure that the system can still run stably under high load conditions, avoiding performance bottlenecks and failure risks.

[0053] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A data processing system for processing a large amount of data, characterized in that: The system consists of the following components: Data receiving module: monitors the data access port, receives data from external data sources, performs preliminary processing on the received data, and stores the processed data for subsequent processing; Data characteristic analysis module: determines the data size through byte counting, identifies the data type, analyzes the source and other identifications of the data, and transmits the analysis results to the intelligent data routing module; Intelligent data routing module: receives data feature information from the data feature analysis module, obtains the real-time load status of the current system from the load balancing module, uses the reinforcement learning model to make routing decisions, sends data to the designated processing node based on the routing decision, and obtains node status information from the processing node cluster in real time to ensure that data can be processed efficiently; Processing node cluster: receives data processing instructions and data from the intelligent data routing module, performs corresponding data processing tasks according to the instructions, and outputs the processing results to the specified location; Load balancing module: Analyzes and calculates the load information of the collection and processing node cluster, evaluates the current system load status, predicts future load peaks based on historical data, and intelligently adjusts data routing strategies to achieve load balancing based on the results of real-time load analysis and predictive analysis.

2. A data processing system for processing a large amount of data according to claim 1, characterized in that: The data receiving module uses a network programming library to listen on a specified port, receives external data sources in the form of byte streams, verifies the integrity of the data, checks the specific identifiers of the data to ensure that the data comes from the expected data source, uses the IQR rule to identify and process outliers, divides the data set into four equal parts of numerical points, Q1 is the value at 25% of the data set, Q3 is the value at 75% of the data set, IQR is the difference between Q3 and Q1, and its mathematical formula is: IQR = Q3 - Q1. The screening condition for outliers based on the IQR rule: x i <Q1 - 1.5 × IQR or x i > Q1 - 1.5 × IQR value, where: x i is each data point in the data set, replaces the identified outliers, normalizes the data, and stores the processed data in the database.

3. A data processing system for processing a large amount of data according to claim 1, characterized in that: The data characteristic analysis module determines the size of the data by calculating the number of bytes of the data, uses a content parsing algorithm to identify the data type, extracts a unique identifier based on the format and content of the data, calculates the arrival rate or flow of the data using exponential smoothing statistics technology to identify network congestion and abnormal behavior, and transmits the analyzed data to the intelligent data routing module according to the characteristics of size, type, source IP address and device ID.

4. A data processing system for processing a large amount of data according to claim 3, characterized in that: The data characteristic analysis module uses exponential smoothing statistical technology to calculate the arrival rate and flow of data, and determines the smoothing coefficient α according to the actual situation. α determines the weight of new data and old data in the prediction. Let S0 be the initial smoothing value, that is, the actual arrival rate of the first period. For each time period t (t=1, 2, 3, ...), the smoothing value calculation formula is: S n =α×y n +(1-α)×S n-1 , where: S n is the smoothed value of period t, y n is the actual arrival rate in period t·S n-1 It is the smoothed value of the t-1th period. The traffic pattern is analyzed by continuously calculating the smoothed values ​​of multiple time periods. A stable smoothed value indicates normal network traffic. A sudden change in the smoothed value indicates network congestion and abnormal behavior. According to historical data and network specifications, normal and abnormal thresholds of the arrival rate are set. If the calculated smoothed value exceeds or falls below these thresholds, an alarm is triggered.

5. A data processing system for processing a large amount of data according to claim 1, characterized in that: The intelligent data routing module receives characteristic information of the data, parses the received characteristic information to understand the nature and processing requirements of the data, obtains the load status, processing capacity, and available resources of the current system from the load balancing module, evaluates the load status of the current system based on the collected load information, and chooses to use the Q-learning model to make routing decisions. According to the routing decision, the data is sent to the designated processing node, and the node status information is obtained from the processing node cluster in real time. Based on the collected node status information, the health status of the processing node is evaluated, and the node status label is updated. If an abnormality is detected in a node, corresponding measures will be taken.

6. A data processing system for processing a large amount of data according to claim 5, characterized in that: The intelligent data routing module uses the Q-learning model to make routing decisions, defines a state set according to the characteristics of the network and processing nodes, determines the action set taken in each state, sends data to a specific processing node, selects a certain routing strategy, initializes a Q value table for each state-action, and sets the Q value to a small random value. It receives characteristic information about the data to be processed from the data characteristic analysis module, obtains the real-time load of the current system from the load balancing module, determines the current state based on the received data characteristics and real-time load, uses the ε-greedy strategy to select an action, randomly selects an action with a probability of ε, selects the action with the maximum Q value in the current state with a probability of 1-ε, routes the data to the corresponding processing node according to the selected action, calculates the reward according to the result after the routing decision, observes and determines the next state after the action is executed, and updates the Q value table using the Q value update formula. The Q value update formula is: Where: Q(s,a) is the Q value of the current state s and action a, α is the learning rate, which determines the degree of coverage of the old Q value by the new information, r is the immediate reward obtained from the environment after executing action a, γ is the discount factor, which is used to balance the importance of immediate rewards and future rewards, A is the set of all actions, s′ is the new state reached after executing action a, It represents the maximum Q value of all actions in the new state s′. Repeat the above steps. Through continuous trial and error and learning, the intelligent data routing module will gradually find the optimal routing strategy.

7. A data processing system for processing a large amount of data according to claim 1, characterized in that: The processing node cluster listens to the data processing instructions and data from the intelligent data routing module through the network interface, parses the data, verifies the integrity, security and format correctness of the data, extracts the data to be processed from the data after the verification, uses the load balancing algorithm to allocate the data processing tasks, processes the data according to the task type application, formats the processing results into a format understood by the intelligent data routing module, outputs them to the specified location, sends its own status information back to the intelligent data routing module in real time, and when a processing node fails, the intelligent data routing module triggers the fault recovery mechanism and reallocates the tasks to other healthy nodes.

8. A data processing system for processing a large amount of data according to claim 1, characterized in that: The load balancing module processes the node load information in real time through a specific API interface and performs standardization on the collected load information so that indicators of different dimensions can be compared. The standardization formula is: Where: l′ ij is the normalized load value, l ij is the original load value, μ j is the mean of the jth index, σ j is the standard deviation of the jth indicator. According to the importance of the indicator and business requirements, a weight is assigned to each indicator, and the weighted sum is used to obtain the comprehensive load value. According to the comprehensive load value of each node, the comprehensive load value calculation formula is: Where: Load i is the comprehensive load value of node i, w j It is the weight of the jth indicator. The average load and load variance are calculated by comprehensive load values, and the load status of the system is analyzed. The ARIMA model is constructed using historical load data to predict the load trend in the future. According to the results of real-time load analysis and predictive analysis, the data routing strategy is adjusted and sent to the intelligent data routing module. The intelligent data routing module performs data routing according to these guidance information to ensure that tasks can be assigned to nodes with lower loads to achieve load balancing.

9. A data processing system for processing a large amount of data according to claim 8, characterized in that: The load balancing module generates a set {h t }, where: t = 1, 2, ..., T, check the stationarity of the data. When the data is not stationary, perform a difference operation until the data becomes stationary. Determine the values ​​of p and q by looking at the graphs of the autocorrelation function ACF and partial autocorrelation function PACF of the differential set. Use historical data to estimate the parameters of the ARIMA model. The ARIMA (p, d, q) model is expressed as: φ (B) (1-B) d h t =θ(B)∈ t , where: φ(B) = 1-φ1B-φ2B 2 -…-φ p B p is an autoregressive polynomial, θ(B) = 1-θ1B-θ2B 2 -…-θ q B q is the moving average polynomial, B is the backshift operator, ∈ t is the error term, which is usually assumed to be a white noise process. Check whether the residual of the model satisfies the white noise assumption. When the model is verified, use the estimated model parameters to predict future load values. T+1 The prediction is expressed as: in: Yes T+1 The predicted value of T+1-j is the error term.

10. A data processing method for processing a large amount of data, characterized in that: The specific steps of this treatment method are: Data reception and verification: Use advanced network programming libraries to listen to designated ports and receive data streams in the form of byte streams. During the reception process, data integrity verification and specific identifier checks are performed to ensure that the received data originates from the expected data source, and outliers are identified and processed, thereby enhancing data security. Data characteristic analysis: Use byte counting, content analysis algorithm and exponential smoothing statistical technology to conduct a comprehensive and in-depth characteristic analysis of the received data. Through this step, the inherent characteristics of the data can be accurately understood, providing rich decision-making basis for subsequent intelligent data routing; Intelligent data routing: Based on the reinforcement learning model, it can analyze data characteristics and the load of the current processing node cluster in real time, dynamically generate the optimal data routing strategy, and ensure that data can be processed efficiently and accurately by designated nodes by intelligently selecting processing nodes, thereby improving the overall processing efficiency of the system; Load balancing optimization: Real-time monitoring of the load status of the processing node cluster, use of prediction algorithms to predict future load peaks, intelligent adjustment of data routing strategies based on prediction results and real-time data analysis, dynamic optimization of load balancing, avoiding performance bottlenecks and failure risks caused by node overload, and stable operation under high load conditions.

Images