A data stream processing system and method in a big data environment

Abstract

The present invention relates to the field of data processing technology, and in particular to a data stream processing system and method in a big data environment. The content includes: real-time data collection to obtain original multi-source heterogeneous data streams; introducing a data adaptation mechanism to perform normalization processing on the original multi-source heterogeneous data streams to obtain normalized data streams; performing intelligent scheduling on normalized data streams to obtain preliminary scheduling strategies; based on the feedback data of the data stream processing system, obtaining parallel real-time status feedback and edge real-time status feedback, optimizing the preliminary scheduling strategy, and based on the optimized scheduling strategy, dividing the normalized data stream into a main normalized data stream and a secondary normalized data stream, and processing the main normalized data stream and the secondary normalized data stream. The problem that the traditional data processing system cannot accurately adapt to multi-source heterogeneous data and the allocation and scheduling are unreasonable when processing data streams in a big data environment is solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data stream processing system and method in a big data environment. Background Art

[0002] In the big data environment, the sources of data are becoming increasingly diverse, including structured data, unstructured data, and semi-structured data. These data usually come from different devices, platforms, or applications, such as sensors, social media, log systems, and real-time streaming data platforms. With the rapid growth of data scale, how to efficiently and accurately collect, process, and analyze data in a multi-source heterogeneous data environment has become an urgent problem to be solved. Traditional data processing systems are usually only applicable to structured or specific format data, and often fail to achieve efficient processing when faced with multi-source heterogeneous data. At the same time, in practical applications, data stream processing places higher requirements on real-time performance and accuracy. With the continuous growth of data volume, traditional batch processing systems have gradually exposed the defects of high latency and inability to meet real-time response requirements.

[0003] In summary, when traditional data processing systems process data streams in a big data environment, they have technical problems such as inability to accurately adapt to multi-source heterogeneous data and unreasonable allocation and scheduling during processing. Summary of the invention

[0004] The present invention provides a data stream processing system and method in a big data environment to solve the technical problems that traditional data processing systems cannot accurately adapt to multi-source heterogeneous data and have unreasonable allocation and scheduling when processing data streams in a big data environment.

[0005] A data stream processing system and method in a big data environment of the present invention specifically includes the following technical solutions:

[0006] A data stream processing method in a big data environment comprises the following steps:

[0007] S1. Collect data in real time to obtain original multi-source heterogeneous data streams; introduce a data adaptation mechanism to normalize the original multi-source heterogeneous data streams to obtain normalized data streams; perform intelligent scheduling on the normalized data streams to obtain a preliminary scheduling strategy;

[0008] S2. Real-time detection of the processing status of the data stream processing system to obtain feedback data; based on the feedback data, obtaining parallel real-time status feedback and edge real-time status feedback; based on the parallel real-time status feedback and edge real-time status feedback, optimizing the preliminary scheduling strategy to obtain an optimized scheduling strategy, and dividing the normalized data stream into a primary normalized data stream and a secondary normalized data stream;

[0009] S3. Based on the optimized scheduling strategy, the primary normalized data stream and the secondary normalized data stream are processed.

[0010] Preferably, the S1 specifically includes:

[0011] The specific implementation steps of the data adaptation mechanism are as follows:

[0012] The first step is to identify and parse the data format in the original multi-source heterogeneous data stream to obtain the parsed multi-source heterogeneous data;

[0013] The second step is to clean and preprocess the parsed multi-source heterogeneous data to obtain preprocessed data;

[0014] The third step is to perform data structure conversion and mapping on the preprocessed data to obtain multi-source data after structure conversion and mapping;

[0015] The fourth step is to perform data type conversion and verification on the multi-source data after structure conversion and mapping, so as to obtain the multi-source data after data type conversion and verification;

[0016] The fifth step is to integrate and output the multi-source data after data type conversion and verification to obtain a standardized data stream.

[0017] Preferably, the S1 specifically includes:

[0018] In the process of data cleaning and preprocessing, a multi-source collaborative anomaly detection algorithm is introduced to perform anomaly detection based on the parsed multi-source heterogeneous data; in the implementation process of the multi-source collaborative anomaly detection algorithm, the parsed multi-source heterogeneous data from different data sources are standardized and converted into vector representation through feature extraction; the similarity measure between data points is calculated in each data source to generate a local similarity matrix.

[0019] Preferably, the S1 specifically includes:

[0020] The local weight coefficient is introduced to weight the similarity measurement between the data points in each data source to obtain the weighted similarity measurement; based on the weighted similarity measurement, the weighted similarity matrix of each data source is obtained; the global weight is introduced to integrate the weighted similarity matrix of each data source to obtain the global integrated similarity matrix.

[0021] Preferably, the S1 specifically includes:

[0022] Define the weight optimization objective function and use the gradient descent method to dynamically optimize the global weight; the specific formula of the weight optimization objective function is as follows:

[0023] ,

[0024] in, represents the weight optimization objective function; Indicates The global weight of each data source; is the total number of data sources; Represents the first data points and The data point in Similarity measure on data sources; Indicates that the parsed multi-source heterogeneous data is The number of data points on each data source; It is the first data points and The data point in The similarity measure obtained after weighting on the data sources represents the weighted similarity measure.

[0025] Preferably, the S1 specifically includes:

[0026] Based on the global comprehensive similarity matrix, the anomaly score of each data point is calculated; a dynamic threshold adjustment mechanism is introduced to obtain a dynamic threshold; the anomaly score of each data point is compared with the dynamic threshold, and when the anomaly score of a data point is greater than the dynamic threshold, it is marked as an anomaly point.

[0027] Preferably, the S2 specifically includes:

[0028] The feedback data includes CPU usage, memory usage, task queue length, network bandwidth and task execution time. Based on the feedback data, the current load of the parallel processing node is calculated by the parallel load calculation formula to obtain parallel real-time status feedback. The parallel load calculation formula is as follows:

[0029] ,

[0030] in, Indicates The current load of the parallel processing nodes, indicating parallel real-time status feedback; They represent the weight parameters used to adjust the effects of CPU usage, memory usage, and task queue length on the total load of the parallel processing node; It is CPU usage of parallel processing nodes; It is Memory usage of parallel processing nodes; It is The length of the task queue of each parallel processing node; It is The maximum capacity of the task queue of a parallel processing node; It is Task execution time of parallel processing nodes; Indicates The maximum allowed execution time for parallel processing nodes;

[0031] Introducing network bandwidth, the current load of the edge computing node is calculated using the edge load calculation formula to obtain real-time edge status feedback; the edge load calculation formula is as follows:

[0032] ,

[0033] in, Indicates The current load of the edge computing node, indicating the real-time status feedback of the edge; It is The utilization rate of network bandwidth of edge computing nodes; It is the parameter weight used to adjust the impact of various factors on the edge computing node load; It is CPU usage of edge computing nodes; It is Memory usage of edge computing nodes; It is The length of the task queue of each edge computing node; It is The maximum capacity of the task queue of an edge computing node; It is The task execution time of each edge computing node; Indicates The maximum allowed execution time of an edge computing node.

[0034] Preferably, the S2 specifically includes:

[0035] The traffic trend of the historical data flow from the existing database is analyzed to obtain the predicted traffic of the data flow; based on the parallel real-time status feedback, the edge real-time status feedback and the predicted traffic of the data flow, the remaining allocatable task volume is calculated; the calculation formula for the remaining allocatable task volume is as follows:

[0036] ,

[0037] in, It is The remaining allocatable tasks of each node; For the The current available resources of each node; For the The current load of each node; It is The predicted traffic share allocated to each node is obtained by allocating the predicted traffic of the data flow according to the remaining resource ratio of each node; It is Average task execution time per node; For the The maximum task execution time scheduled by a node.

[0038] Preferably, the S2 specifically includes:

[0039] Based on the remaining allocatable tasks of the node, the tasks are dynamically adjusted through the global load balancing formula to obtain the optimized scheduling strategy; the global load balancing formula is as follows:

[0040] ,

[0041] in, It is The optimized task allocation of each node represents the optimized scheduling strategy; It is The initial allocation of tasks for each node; is the number of all nodes; It is The initial allocation of tasks for each node; It is The remaining allocatable tasks of each node; It is The remaining allocatable tasks of each node; is The predicted flow of data flow at that time;

[0042] Based on the optimized scheduling strategy, tasks are reallocated to parallel processing nodes and edge computing nodes, and adjustments and feedback are made to form a closed-loop scheduling mechanism.

[0043] A data stream processing system in a big data environment includes the following parts:

[0044] Data acquisition module, heterogeneous data adaptation module, intelligent scheduling module, parallel processing module, edge computing module;

[0045] The data acquisition module collects data in real time to obtain the original multi-source heterogeneous data stream; the original multi-source heterogeneous data stream is sent to the heterogeneous data adaptation module;

[0046] The heterogeneous data adaptation module introduces a data adaptation mechanism to normalize the original multi-source heterogeneous data stream to obtain a normalized data stream; the normalized data stream is sent to the intelligent scheduling module; based on the optimized scheduling strategy from the intelligent scheduling module and combined with the normalized data stream characteristics, the normalized data stream is divided into a main normalized data stream and a secondary normalized data stream; the main normalized data stream and the secondary normalized data stream are sent to the parallel processing module and the edge computing module respectively;

[0047] The intelligent scheduling module intelligently schedules the normalized data stream to obtain a preliminary scheduling strategy; based on the parallel real-time state feedback of the parallel processing module and the edge real-time state feedback of the edge computing module, as well as the predicted flow of the data stream, the preliminary scheduling strategy is optimized to obtain an optimized scheduling strategy; the optimized scheduling strategy is sent to the parallel processing module, the edge computing module and the heterogeneous data adaptation module;

[0048] The parallel processing module, based on the optimized scheduling strategy, performs independent parallel processing on the main normalized data stream through the task allocation algorithm, and detects the processing status of the parallel processing module in real time; and sends the parallel real-time status feedback to the intelligent scheduling module;

[0049] The edge computing module performs edge processing on the sub-normalized data stream based on the optimized scheduling strategy, and detects the processing status of the edge computing module in real time; and sends the edge real-time status feedback to the intelligent scheduling module.

[0050] The beneficial effects of the technical solution of the present invention are:

[0051] 1. By introducing a data adaptation mechanism, it can effectively cope with multi-source heterogeneous data streams of different sources, formats and types, ensuring the compatibility and efficient parsing capabilities of the data stream processing system for multiple data sources. The multi-source collaborative anomaly detection algorithm is introduced to analyze the similarities between data points in multiple dimensions (such as source, format, time and contextual relationship), effectively improving the accuracy of identifying abnormal data. The multi-source collaborative anomaly detection algorithm constructs a weighted similarity matrix and a global comprehensive similarity matrix, optimizes the global weights of different data sources, and introduces a dynamic threshold adjustment mechanism to make anomaly detection more sensitive, solving the problem that traditional data stream processing methods cannot effectively detect abnormal patterns when processing multi-source heterogeneous data, and significantly improving the effect of data cleaning.

[0052] 2. Through the real-time feedback mechanism of parallel processing and edge computing, combined with the predicted traffic of data streams, the data stream processing system can achieve continuous optimization of the scheduling strategy. The introduction of the load calculation formula enables the data stream processing system to evaluate the load status of each node in real time during operation, and dynamically adjust the allocation of resources and task scheduling to generate an optimized scheduling strategy; based on the optimized scheduling strategy, tasks are reallocated to the corresponding parallel processing nodes or edge computing nodes, and continuous adjustments and feedback are made according to the actual execution situation to form a closed-loop scheduling mechanism. The closed-loop scheduling mechanism ensures that the data stream processing system always maintains the best processing efficiency when dealing with data traffic fluctuations.

[0053] 3. Through the task allocation algorithm based on the optimized scheduling strategy, the data stream processing system can balance the load of each node to avoid the situation where some nodes are overloaded while other nodes are idle; in particular, the minimum load priority algorithm can dynamically monitor the load of each node to ensure the balance of the overall resource utilization of the data stream processing system, further improving the processing capacity of the data stream processing system, especially during peak data traffic periods, the data stream processing system can still maintain efficient operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a structural diagram of a data stream processing system in a big data environment according to the present invention;

[0055] Figure 2 This is a flow chart of a data stream processing method in a big data environment described in the present invention. DETAILED DESCRIPTION

[0056] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0058] The following describes in detail a specific scheme of a data stream processing system and method in a big data environment provided by the present invention in conjunction with the accompanying drawings.

[0059] See attached Figure 1 , which shows a structure diagram of a data stream processing system in a big data environment provided by an embodiment of the present invention, the system includes the following parts:

[0060] Data acquisition module, heterogeneous data adaptation module, intelligent scheduling module, parallel processing module, edge computing module;

[0061] The data acquisition module collects data from different sources in real time to obtain the original multi-source heterogeneous data stream; the original multi-source heterogeneous data stream is sent to the heterogeneous data adaptation module;

[0062] The heterogeneous data adaptation module introduces a data adaptation mechanism to normalize the original multi-source heterogeneous data stream to obtain a normalized data stream; the normalized data stream is sent to the intelligent scheduling module; based on the optimized scheduling strategy from the intelligent scheduling module and combined with the normalized data stream characteristics, the normalized data stream is divided into a main normalized data stream and a secondary normalized data stream; the main normalized data stream and the secondary normalized data stream are sent to the parallel processing module and the edge computing module respectively;

[0063] The data adaptation mechanism includes data format identification and analysis, data cleaning and preprocessing, data structure conversion and mapping, data type conversion and verification, data integration and output;

[0064] The intelligent scheduling module intelligently schedules the normalized data stream in combination with the status information of the current data stream processing system to obtain a preliminary scheduling strategy; based on the parallel real-time status feedback of the parallel processing module and the edge real-time status feedback of the edge computing module, as well as the predicted flow of the data stream, the preliminary scheduling strategy is optimized to obtain an optimized scheduling strategy; the optimized scheduling strategy is sent to the parallel processing module, the edge computing module and the heterogeneous data adaptation module;

[0065] The parallel processing module, based on the optimized scheduling strategy, divides the main normalized data stream into multiple task slices through the task allocation algorithm, realizes independent parallel processing on the parallel processing nodes, and detects the processing status of the parallel processing module in real time; and sends the parallel real-time status feedback to the intelligent scheduling module;

[0066] The edge computing module, based on the optimized scheduling strategy, performs edge processing on the sub-normalized data stream on the edge device close to the data source, and detects the processing status of the edge computing module in real time; and sends the edge real-time status feedback to the intelligent scheduling module.

[0067] See attached Figure 2 , which shows a flow chart of a data stream processing method in a big data environment provided by an embodiment of the present invention, the method comprising the following steps:

[0068] S1. Collect data in real time to obtain original multi-source heterogeneous data streams; introduce a data adaptation mechanism to normalize the original multi-source heterogeneous data streams to obtain normalized data streams; perform intelligent scheduling on the normalized data streams to obtain a preliminary scheduling strategy;

[0069] Collect data from different sources in real time to obtain original multi-source heterogeneous data streams, including structured data (such as databases), unstructured data (such as social media content, log files), and semi-structured data (such as JSON, XML, etc.);

[0070] Furthermore, a data adaptation mechanism is introduced to uniformly normalize the original multi-source heterogeneous data streams to obtain a normalized data stream; the specific contents are:

[0071] The first step is data format identification and parsing. The data in the original multi-source heterogeneous data stream is formatted to ensure that different types of data sources can be correctly parsed and processed. First, regular expressions, MIME type detection, or file header identifiers are used to automatically identify the data format of the original multi-source heterogeneous data. Common data formats include JSON, XML, CSV, YAML, text, binary files, etc. Specifically, for structured data (such as database tables, CSV files), the data format can be quickly identified through file extensions and file header features; for unstructured data (such as logs, text files), the format can be identified through file signatures, data structure rules or regular expressions; for streaming data (such as real-time sensor data or API response data), the data format can be inferred based on the transmission protocol and data type; after the format identification is completed, according to the data format combined with expert experience, select appropriate parsing tools to extract data content to obtain parsed multi-source heterogeneous data; for example: select a JSON parser (such as Gson, Jackson) to parse data in JSON format; select an XML parser (such as DOM, SAX, XPath) to process XML data; select a CSV parser (such as OpenCSV) to process CSV files; select a log parser to extract valid information from unstructured text (such as using Logstash to parse log data).

[0072] The second step is data cleaning and preprocessing. In the process of processing the data in the original multi-source heterogeneous data stream, incomplete, redundant, erroneous or noisy data are often encountered. Therefore, it is necessary to clean and preprocess the parsed multi-source heterogeneous data to obtain preprocessed data. The cleaning and preprocessing steps include missing data processing, duplicate data cleaning and abnormal data processing.

[0073] Among them, in the process of cleaning and preprocessing the parsed multi-source heterogeneous data, due to the differences between data of different sources, formats, and types, traditional methods cannot effectively detect abnormal patterns in the parsed multi-source heterogeneous data, resulting in low accuracy in identifying abnormal data. Therefore, a multi-source collaborative anomaly detection algorithm is introduced to perform multi-level analysis and detection and disassembly of the parsed multi-source heterogeneous data through multiple dimensions (source, format, time, and contextual relationship) based on the similarity and correlation of data from different sources.

[0074] The specific implementation process of the multi-source collaborative anomaly detection algorithm is as follows:

[0075] For the parsed multi-source heterogeneous data, the data from different data sources are first standardized and converted into vector representation through feature extraction. Different data sources (e.g. sensor data, social media data, log data, etc.) are used to convert each data point Represented as a dimensional feature vector:

[0076] ,

[0077] in, Represents the first The data point in Feature vectors on data sources; It is the first The data point in The first eigenvalues; It is the first The data point in The first eigenvalues;

[0078] Furthermore, in order to evaluate the similarity between data points in the parsed multi-source heterogeneous data, a local similarity matrix is ​​constructed in each data source. The Gaussian kernel function (Radial Basis Function, RBF) is used to measure the similarity. Any element in the local similarity matrix can be expressed as:

[0079] ,

[0080] in, Represents the first data points and The data point in Similarity measure on data sources; Represents the first The data point in Feature vectors on data sources; is the Euclidean distance, which is used to measure the first data points and The distance between data points; It is The sensitivity parameter of each data source is used to control the rate of change of the similarity measure, which can be adaptively adjusted according to the fluctuation of the data source;

[0081] In order to further refine the similarity measurement, the local weight coefficient is introduced. The local similarity matrix is ​​weighted according to the importance of the data source to obtain the weighted similarity matrix of each data source. Any element in the weighted similarity matrix can be expressed as:

[0082] ,

[0083] in, It is determined by expert experience based on factors such as data source quality, credibility, and historical performance. The local weight coefficient of each data source; It is the first data points and The data point in The weighted similarity measure obtained on the data sources is the weighted similarity measure; a weighted similarity matrix is ​​constructed based on the weighted similarity measure;

[0084] After obtaining the weighted similarity matrix of each data source, the weighted similarity matrix of each data source is integrated by linear weighted summation to construct a global comprehensive similarity matrix. Any element in can be represented as:

[0085] ,

[0086] in, It is The global weight of each data source is used to control the contribution of different data sources to the final similarity; After integrating various data sources, data points and The global similarity of data points.

[0087] In order to further optimize the global weights of different data sources, the weight optimization objective function is defined , used to balance the consistency of similarity and the performance of data sources:

[0088] ,

[0089] in, Indicates that the parsed multi-source heterogeneous data is The weight optimization objective function aims to maximize the joint consistency of the similarities of each data source, and the Laplace regularization term is used to ensure the stability of the optimization process. Find the partial derivative and use the gradient descent method to dynamically optimize the Global weight of data sources The value of makes the global weight change over time; in each iteration, the objective function is optimized based on the weight The gradient update rule is:

[0090] ,

[0091] in, It is After the iteration, the The global weight of each data source, that is, the optimized global weight; It is After the iteration, the The global weight of the data source, that is, the current global weight; is the learning rate; Optimize the objective function for the weights For the current global weight The partial derivative of represents the current global weight The gradient of indicates the adjustment direction of the current global weight.

[0092] After constructing the global comprehensive similarity matrix After that, the anomaly score of each data point is calculated; a data point with a high anomaly score indicates that the data point has a low similarity with other data points and may be an outlier. In order to evaluate the overall deviation of each data point from other data points, the following anomaly score calculation formula is introduced:

[0093] ,

[0094] in, Indicates Data points Anomaly score of Represents the total number of data points of the parsed multi-source heterogeneous data; is a distance correction function, which is used to further enhance the effect of similarity on the anomaly score. The specific formula of the distance correction function is as follows:

[0095] ,

[0096] in, To adjust the parameters, used to control the sensitivity of distance correction; Indicates data points; It is Data points The standard deviation of .

[0097] After calculating the anomaly score of each data point, in order to improve the sensitivity of anomaly detection, a dynamic threshold adjustment mechanism is introduced. The distribution of anomaly scores is dynamically adjusted using sliding window technology. A sliding window anomaly score sequence is dynamically extracted and the mean of the sliding window anomaly score sequence is calculated. and standard deviation , further calculate the dynamic threshold :

[0098] ,

[0099] in, is the mean of the anomaly score sequence in the current window; is the standard deviation of the anomaly score sequence within the current window; is an adjustment factor used to control the flexibility of the dynamic threshold.

[0100] Finally, based on the dynamic threshold , for each data point, if , then the data point is marked as an outlier; otherwise, the data point is considered a normal point.

[0101] The third step is data structure conversion and mapping. After data cleaning and preprocessing, due to the large differences in the data structures of multi-source heterogeneous data, it is necessary to uniformly convert various data structures into a system standard data model. First, the existing standardization technology is used to standardize the data structure of the preprocessed data (such as flattening the nested structure and field standardization) to obtain standardized data; further, the standardized data is converted and mapped using existing technical means, such as defining a standardized target data model for data from different data sources to ensure that data from different data sources can be mapped to a unified structure. Tools such as JSON Schema and Avro that describe data structures can be used to define and constrain the format of the preprocessed data to obtain structured data; then the structured data is associated and merged with multiple tables to finally obtain multi-source data after structure conversion and mapping.

[0102] The fourth step is data type conversion and verification. In order to ensure that the multi-source data after structural conversion and mapping remains consistent during the processing, the data types of different data sources are converted to a unified standard type, such as converting "timestamp" format data to a unified timestamp format (such as ISO 8601 format), and converting string type numbers to numeric types (such as converting "123" to integer 123); standardize the measurement units in the multi-source data after structural conversion and mapping. At the same time, after the data type conversion, perform data integrity verification (such as range verification, format verification, and data dependency verification) to obtain the multi-source data after data type conversion and verification.

[0103] The fifth step is data integration and output. Integrate and output the multi-source data after data type conversion and verification for subsequent processing. Use existing technologies such as batch processing, stream processing, and data aggregation to merge and obtain an integrated data stream, that is, a standardized data stream, which provides a data basis for subsequent data processing.

[0104] After the above processing, a normalized data stream is obtained.

[0105] Before processing the normalized data stream, based on the status information of the current data stream processing system (such as CPU, memory, disk I / O and network bandwidth usage), a scheduling algorithm suitable for the processing scenario is selected according to expert experience to perform intelligent scheduling on the normalized data stream to obtain a preliminary scheduling strategy; the scheduling algorithm includes polling scheduling, weighted polling scheduling, minimum task queue algorithm, data stream priority scheduling algorithm, and prediction-based scheduling algorithm.

[0106] When performing intelligent scheduling, feature analysis is performed on the normalized data stream. Different data streams differ in data volume, data processing complexity, real-time performance, priority, etc. It is necessary to perform feature extraction and analysis on each normalized data stream in order to provide a basis for subsequent scheduling. Here, feature extraction and analysis are performed using means familiar to technical personnel in this field.

[0107] S2. Real-time detection of the processing status of the data stream processing system to obtain feedback data; based on the feedback data, obtaining parallel real-time status feedback and edge real-time status feedback; based on the parallel real-time status feedback and edge real-time status feedback, optimizing the preliminary scheduling strategy to obtain an optimized scheduling strategy, and dividing the normalized data stream into a primary normalized data stream and a secondary normalized data stream;

[0108] In the process of optimizing the preliminary scheduling strategy, it is necessary to conduct a comprehensive analysis through real-time feedback from parallel processing and edge computing and predicted traffic of data flow to ensure reasonable allocation of resources, avoid system overload or waste of resources, and dynamically adapt to fluctuations in data traffic. The specific implementation steps for optimizing the preliminary scheduling strategy are as follows:

[0109] First, the processing status of the data stream processing system is detected in real time to obtain feedback data from the parallel processing module and the edge computing module;

[0110] The feedback data of the parallel processing module and edge computing module include CPU utilization, memory utilization, task queue length, network bandwidth, and task execution time.

[0111] Based on the feedback data, the current load of the parallel processing node is calculated by the parallel load calculation formula to obtain parallel real-time state feedback to evaluate the current load of the parallel processing node. The parallel load calculation formula of any parallel processing node can be expressed as:

[0112] ,

[0113] in, Indicates The current load of a parallel processing node represents the resource usage of the current parallel processing node and is an important indicator for measuring whether the parallel processing node can continue to receive tasks, that is, parallel real-time status feedback; They are weight parameters, which are used to adjust the impact of CPU usage, memory usage, and task queue length on the total load of the parallel processing node, and are set according to the actual application scenario and the characteristics of the parallel processing node; It is The CPU usage of a parallel processing node is used to indicate the CPU resource usage of the current parallel processing node. The value range is 0 to 1, where 0 represents idle and 1 represents full load. It is The memory usage of a parallel processing node is used to indicate the memory resource usage of the current parallel processing node. The value range is 0 to 1, where 0 represents no memory usage and 1 represents full memory usage. It is The task queue length of a parallel processing node (Queue Length) indicates how many tasks are waiting to be executed on the current parallel processing node. The task queue length reflects the task backlog of the parallel processing node; It is The maximum capacity of the task queue of a parallel processing node, which indicates the upper limit of the number of tasks that the current parallel processing node can carry, is used to calculate the ratio of the current task queue to the total capacity; It is The task execution time (Execution Time) of a parallel processing node indicates the average execution time of the current task on the current parallel processing node; Indicates The maximum allowed execution time of a parallel processing node indicates the upper limit of the task execution time of the current parallel processing node, which is used to determine whether the task execution has timed out or is too long. The denominator of the parallel load calculation formula introduces the influencing parameter of the execution time. ,When the task execution time is too long, reduce the load distribution of the parallel processing node.

[0114] Since edge computing involves data transmission, it is necessary to introduce the factor of network bandwidth to obtain real-time edge status feedback; the formula for calculating the edge load of any edge computing node is as follows:

[0115] ,

[0116] in, Indicates The current load of each edge computing node, i.e., real-time edge status feedback; It is The network bandwidth usage of each edge computing node, ranging from 0 to 1; It is the weight of the corresponding parameter, which is used to adjust the impact of various factors on the load of edge computing nodes; It is The CPU usage of an edge computing node is used to indicate the CPU resource usage of the current edge computing node. The value range is 0 to 1, where 0 represents idle and 1 represents full load. It is The memory usage of an edge computing node indicates the memory resource usage of the current edge computing node. The value range is 0 to 1, where 0 means that the memory is not occupied and 1 means that the memory is full. It is The task queue length (Queue Length) of an edge computing node indicates how many tasks are waiting to be executed on the current edge computing node. The task queue length reflects the task backlog of the current edge computing node; It is The maximum capacity of the task queue of an edge computing node, which indicates the upper limit of the number of tasks that the current edge computing node can carry, is used to calculate the ratio of the current task queue to the total capacity; It is The task execution time (Execution Time) of an edge computing node indicates the average execution time of the current task on the current edge computing node; Indicates The maximum allowed execution time of an edge computing node indicates the upper limit of the task execution time of the current edge computing node, which is used to determine whether the task execution has timed out or is too long. The particularity of edge load calculation is that its network bandwidth usage will also significantly affect the edge computing node load, so the network bandwidth is added to the weight calculation to reflect the actual status.

[0117] Use the existing ARIMA model or LSTM model to build models and analyze the traffic trends of historical data flows in existing databases to predict The traffic volume of the data flow at that time is used to obtain the predicted traffic volume of the data flow .

[0118] Furthermore, the resource allocation strategy is dynamically adjusted by combining parallel real-time status feedback, edge real-time status feedback, and predicted data flow. To this end, the remaining allocatable tasks of each node (parallel processing node or edge computing node) are first calculated based on its current load. The calculation formula for the remaining allocatable tasks is as follows:

[0119] ,

[0120] in, It is The remaining allocatable tasks of each node; For the The current available resources of each node; For the The current load of each node; It is The predicted traffic share allocated to each node is the predicted traffic share of the data flow according to the remaining resource ratio of each node. The remaining resource ratio is a known parameter. It is Average task execution time per node; For the The maximum task execution time scheduled by each node. The calculation formula for the remaining task amount is based on The current load of each node and the predicted flow of the data flow are used to calculate the remaining assignable tasks of the node.

[0121] Furthermore, in order to ensure the load balance of the data stream processing system, the load status of all nodes needs to be summarized into a global scheduling decision. On this basis, the tasks are redistributed through the global load balancing formula:

[0122] ,

[0123] in, It is The optimized task allocation of each node represents the optimized scheduling strategy; It is The initial allocation of tasks for each node; is the number of all nodes (including parallel processing nodes and edge computing nodes); It is The initial allocation of tasks for each node; It is The remaining allocatable tasks of each node; It is The remaining allocatable tasks of each node are calculated based on the parallel real-time status feedback of the parallel processing node and the current load in the edge real-time status feedback of the edge computing node;

[0124] The current load calculation formula of the parallel processing node is:

[0125] ,

[0126] The current load calculation formula of the edge computing node is:

[0127] ,

[0128] It is The current available resources of each node; It is The predicted traffic share allocated to each node; It is Average task execution time per node; For the The maximum execution time scheduled by a node; It is The current load of the parallel processing nodes; It is The current load of edge computing nodes;

[0129] The global load balancing formula dynamically adjusts tasks according to the remaining allocatable tasks of each node and the predicted traffic of the data flow to generate an optimized scheduling strategy; based on the optimized scheduling strategy, tasks are reallocated to the corresponding parallel processing nodes or edge computing nodes, and continuous adjustments and feedback are made based on the actual execution situation to form a closed-loop scheduling mechanism.

[0130] The normalized data stream is divided into a main normalized data stream and a sub-normalized data stream according to the optimized scheduling strategy and the characteristics of the normalized data stream (size, priority, low latency requirement or high real-time requirement) based on expert experience, and sent to the parallel processing module and the edge computing module respectively; the main normalized data stream and the sub-normalized data stream do not represent the importance of the normalized data stream.

[0131] S3. Based on the optimized scheduling strategy, the primary normalized data stream and the secondary normalized data stream are processed.

[0132] Based on the optimized scheduling strategy, the task allocation algorithm is selected according to the application scenario, and the main normalized data stream is divided into multiple task slices and allocated to different parallel processing nodes to achieve independent parallel processing; the task allocation algorithm includes hash sharding algorithm, consistent hashing algorithm, weighted polling algorithm, and minimum load first algorithm.

[0133] At the same time, based on the optimized scheduling strategy, the sub-normalized data stream is partially processed on the edge device close to the data source to realize data stream processing.

[0134] In summary, a data stream processing system and method in a big data environment are completed.

[0135] The order of the embodiments of the invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0136] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0137] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A data stream processing method in a big data environment, characterized in that: The following steps are involved: S1. Collect data in real time to obtain original multi-source heterogeneous data streams; introduce a data adaptation mechanism to normalize the original multi-source heterogeneous data streams to obtain normalized data streams; perform intelligent scheduling on the normalized data streams to obtain a preliminary scheduling strategy; S2. Real-time detection of the processing status of the data stream processing system to obtain feedback data; the feedback data includes CPU usage, memory usage, task queue length, network bandwidth and task execution time; based on the feedback data, calculate parallel real-time status feedback, the specific formula is as follows: Among them, LP s represents the current load of the sth parallel processing node, and represents the parallel real-time status feedback; ω1, ω2, ω3 represent the weight parameters used to adjust the influence of CPU utilization, memory utilization, and task queue length on the total load of parallel processing nodes; CPU s is the CPU usage of the sth parallel processing node; MEM s is the memory usage of the sth parallel processing node; QL s is the task queue length of the sth parallel processing node; Max QLs is the maximum capacity of the task queue of the sth parallel processing node; ET s is the task execution time of the sth parallel processing node; ET s,max represents the maximum allowed execution time of the sth parallel processing node; Introducing network bandwidth, calculating edge real-time status feedback, the specific formula is as follows: Among them, LE l Indicates the current load of the lth edge computing node, indicating the real-time status feedback of the edge; BW l is the utilization rate of the network bandwidth of the lth edge computing node; It is the parameter weight used to adjust the impact of various factors on the edge computing node load; CPU l is the CPU usage of the lth edge computing node; MEM l is the memory usage of the lth edge computing node; QL l is the task queue length of the lth edge computing node; is the maximum capacity of the task queue of the lth edge computing node; ET l is the task execution time of the lth edge computing node; ET l,max represents the maximum allowed execution time of the lth edge computing node; Based on parallel real-time state feedback and edge real-time state feedback, the preliminary scheduling strategy is optimized to obtain the optimized scheduling strategy, and the normalized data stream is divided into a main normalized data stream and a sub-normalized data stream; S3. Based on the optimized scheduling strategy, the primary normalized data stream and the secondary normalized data stream are processed.

2. The data stream processing method in a big data environment according to claim 1 is characterized in that: The S1 specifically includes: The specific implementation steps of the data adaptation mechanism are as follows: The first step is to identify and parse the data format in the original multi-source heterogeneous data stream to obtain the parsed multi-source heterogeneous data; The second step is to clean and preprocess the parsed multi-source heterogeneous data to obtain preprocessed data; The third step is to perform data structure conversion and mapping on the preprocessed data to obtain multi-source data after structure conversion and mapping; The fourth step is to perform data type conversion and verification on the multi-source data after structure conversion and mapping, so as to obtain the multi-source data after data type conversion and verification; The fifth step is to integrate and output the multi-source data after data type conversion and verification to obtain a standardized data stream.

3. The data stream processing method in a big data environment according to claim 2 is characterized in that: The S1 specifically includes: In the process of data cleaning and preprocessing, a multi-source collaborative anomaly detection algorithm is introduced to perform anomaly detection based on the parsed multi-source heterogeneous data; in the implementation process of the multi-source collaborative anomaly detection algorithm, the parsed multi-source heterogeneous data from different data sources are standardized and converted into vector representation through feature extraction; the similarity measure between data points is calculated in each data source to generate a local similarity matrix.

4. The data stream processing method in a big data environment according to claim 3 is characterized in that: The S1 specifically includes: The local weight coefficient is introduced to weight the similarity measurement between the data points in each data source to obtain the weighted similarity measurement; based on the weighted similarity measurement, the weighted similarity matrix of each data source is obtained; the global weight is introduced to integrate the weighted similarity matrix of each data source to obtain the global integrated similarity matrix.

5. The data stream processing method in a big data environment according to claim 4 is characterized in that: The S1 specifically includes: Define the weight optimization objective function and use the gradient descent method to dynamically optimize the global weight; the specific formula of the weight optimization objective function is as follows: in, represents the weight optimization objective function; α k represents the global weight of the kth data source; m is the total number of data sources; Represents the i-th data point and the i-th data point in the parsed multi-source heterogeneous data The similarity measure of data points on the kth data source; Represents the number of data points of the parsed multi-source heterogeneous data on the kth data source; is the i-th data point and the i-th data point in the parsed multi-source heterogeneous data. The similarity measure obtained by weighting the data points on the kth data source represents the weighted similarity measure.

6. The data stream processing method in a big data environment according to claim 4 is characterized in that: The S1 specifically includes: Based on the global comprehensive similarity matrix, the anomaly score of each data point is calculated; a dynamic threshold adjustment mechanism is introduced to obtain a dynamic threshold; the anomaly score of each data point is compared with the dynamic threshold, and when the anomaly score of a data point is greater than the dynamic threshold, it is marked as an anomaly point.

7. The data stream processing method in a big data environment according to claim 1 is characterized in that: The S2 specifically includes: The traffic trend of the historical data flow from the existing database is analyzed to obtain the predicted traffic of the data flow; based on the parallel real-time status feedback, the edge real-time status feedback and the predicted traffic of the data flow, the remaining allocatable task volume is calculated; the calculation formula for the remaining allocatable task volume is as follows: Among them, T rem,q is the remaining allocatable tasks of the qth node; R avail,q is the current available resource amount of the qth node; Load current,q is the current load of the qth node; F t+1,q is the predicted traffic share allocated to the qth node, which is obtained by allocating the predicted traffic of the data flow according to the remaining resource ratio of each node; ET avg,q is the average task execution time of the qth node; ET max,q The maximum task execution time scheduled for the qth node.

8. The data stream processing method in a big data environment according to claim 7 is characterized in that: The S2 specifically includes: Based on the remaining allocatable tasks of the node, the tasks are dynamically adjusted through the global load balancing formula to obtain the optimized scheduling strategy; the global load balancing formula is as follows: Among them, T opt,p is the optimized task allocation of the pth node, indicating the optimized scheduling strategy; T init,p is the initial assigned task amount of the pth node; is the number of all nodes; It is The initial allocation of tasks for each node; It is The remaining allocatable tasks of each node; T rem,p is the remaining allocatable tasks of the pth node; is the predicted flow rate of the data stream at t+1; Based on the optimized scheduling strategy, tasks are reallocated to parallel processing nodes and edge computing nodes, and adjustments and feedback are made to form a closed-loop scheduling mechanism.

9. A data stream processing system in a big data environment, applied to a data stream processing method in a big data environment as claimed in claim 1, characterized in that: Includes the following sections: Data acquisition module, heterogeneous data adaptation module, intelligent scheduling module, parallel processing module, edge computing module; The data acquisition module collects data in real time to obtain the original multi-source heterogeneous data stream; the original multi-source heterogeneous data stream is sent to the heterogeneous data adaptation module; The heterogeneous data adaptation module introduces a data adaptation mechanism to normalize the original multi-source heterogeneous data stream to obtain a normalized data stream; the normalized data stream is sent to the intelligent scheduling module; Based on the optimized scheduling strategy from the intelligent scheduling module and combined with the normalized data stream characteristics, the normalized data stream is divided into a main normalized data stream and a sub-normalized data stream; the main normalized data stream and the sub-normalized data stream are sent to the parallel processing module and the edge computing module respectively; The intelligent scheduling module performs intelligent scheduling on the standardized data flow to obtain a preliminary scheduling strategy; based on the parallel real-time state feedback of the parallel processing module and the edge real-time state feedback of the edge computing module, as well as the predicted flow of the data flow, the preliminary scheduling strategy is optimized to obtain an optimized scheduling strategy; Send the optimized scheduling strategy to the parallel processing module, edge computing module and heterogeneous data adaptation module; The parallel processing module, based on the optimized scheduling strategy, performs independent parallel processing on the main normalized data stream through the task allocation algorithm and detects the processing status of the parallel processing module in real time; Send parallel real-time status feedback to the intelligent scheduling module; The edge computing module performs edge processing on the sub-normalized data stream based on the optimized scheduling strategy and detects the processing status of the edge computing module in real time; Send edge real-time status feedback to the intelligent scheduling module.

Images