An efficient big data processing system based on a distributed computing architecture

Abstract

The application discloses a kind of efficient big data processing systems based on distributed computing architecture, it is related to distributed computing and big data processing technical field, the system includes following component parts: space-time feature extraction module, space-time prediction engine, resource pre-scheduling center, space-time perception routing module and flow batch collaborative processing engine;The application is through resource pre-scheduling center, using space-time sensitivity weighted scheduling model and cache pool dynamic adjustment unit, in advance and accurately allocate computing, storage and bandwidth resources, specifically, the computing power pre-allocation unit is according to the spatial sensitivity of region, time sensitivity and data growth rate dynamically adjusts computing power allocation, effectively avoids resource waste, and ensures that high sensitivity area is in peak period Resource is sufficient, to significantly improve the overall resource utilization rate, at the same time, the cache pool dynamic adjustment unit realizes the accurate allocation of cache resources by fusing predicted data volume, node load and region relevance.

Description

Technical Field

[0001] This invention relates to the field of distributed computing and big data processing technology, specifically to a high-efficiency big data processing system based on a distributed computing architecture. Background Technology

[0002] With the rapid development of Internet technology and the explosive growth of data volume, big data processing has become one of the core challenges in the current information technology field. In a distributed computing environment, how to efficiently process and analyze massive amounts of data has become the key to improving business decision-making capabilities and user experience.

[0003] Traditional big data processing technologies rely primarily on static resource allocation strategies and fixed data processing workflows, making it difficult to adapt to dynamically changing data environments and business needs. Specifically, when processing large-scale data streams, these systems often face problems such as uneven resource allocation, unbalanced computing node loads, and low data transmission efficiency. On the one hand, due to a lack of in-depth analysis and prediction of the spatiotemporal characteristics of data, traditional systems struggle to accurately allocate computing, storage, and bandwidth resources in advance, leading to frequent resource waste or shortages. On the other hand, the fixed time window processing method results in a severe spatiotemporal mismatch between stream processing and batch processing, affecting the accuracy and real-time performance of data processing. Furthermore, traditional routing algorithms often neglect key factors such as spatial correlation and historical collaborative processing success rates when optimizing data transmission paths, resulting in persistently high overall transmission costs.

[0004] In response to the problems of low resource utilization, low processing efficiency and high data transmission costs in traditional big data processing technologies, this invention proposes a highly efficient big data processing system based on a distributed computing architecture, which is of particular importance. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a high-efficiency big data processing system based on a distributed computing architecture. By integrating multiple core components such as a spatiotemporal feature extraction module, a spatiotemporal prediction engine, a resource pre-scheduling center, a spatiotemporal-aware routing module, and a stream-batch collaborative processing engine, it achieves accurate prediction of data streams, dynamic allocation of resources, optimization of transmission paths, and accurate fusion of stream and batch processing results. This system can significantly improve resource utilization and data processing efficiency, reduce overall transmission costs, and provide strong technical support for modern big data processing.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a high-efficiency big data processing system based on a distributed computing architecture, the system comprising the following components: a spatiotemporal feature extraction module, a spatiotemporal prediction engine, a resource pre-scheduling center, a spatiotemporal-aware routing module, and a stream-batch collaborative processing engine;

[0007] The spatiotemporal feature extraction module: collects spatiotemporal labels of the data stream and extracts spatiotemporal feature vectors, wherein the spatiotemporal feature vectors include dynamic spatiotemporal correlation degree;

[0008] The spatiotemporal prediction engine predicts the spatiotemporal distribution of future data streams based on the spatiotemporal feature vectors and outputs a prediction table. The spatiotemporal prediction engine includes a time series prediction submodule, a spatial correlation submodule, and a prediction result correction module.

[0009] The resource pre-scheduling center: allocates computing, storage and bandwidth resources in advance according to the prediction table and node resource status. The resource pre-scheduling center includes a computing power pre-allocation unit and a cache pool dynamic adjustment unit.

[0010] The spatiotemporal-aware routing module optimizes the data transmission path based on the predicted spatiotemporal distribution and calculates the path cost using a spatiotemporal cost-optimal path algorithm.

[0011] The stream-batch collaborative processing engine unifies the time windows for stream processing and batch processing to achieve collaborative data processing. The stream-batch collaborative processing engine includes a time window alignment unit and a result fusion unit.

[0012] Furthermore, the dynamic spatiotemporal correlation degree extracted by the spatiotemporal feature extraction module is obtained through the following algorithm: Let a certain region In the time window The internal data stream is ,area With the region The spatial distance is Time window With historical window The time difference is Then the dynamic spatiotemporal correlation degree The calculation formula is: ,in For the region and In time The intersection of data streams within, For the union, , , Weighting coefficients , , ,and Its value is determined by training samples of highly correlated regions in historical data. Regions that have experienced simultaneous outbreaks in the past 3 months are selected as positive samples, and those that have not are selected as negative samples. The gradient descent method is used to optimize the weights to minimize the correlation prediction error. The spatial reference distance is taken as 1.5 times the average distance between areas within the system's coverage area. The time base window is set at 2 hours, determined based on statistical analysis of data burst cycles in most scenarios.

[0013] Furthermore, the temporal prediction submodule of the spatiotemporal prediction engine employs a dynamically decaying LSTM algorithm, and its cell state update formula is as follows: ,in Current time The state of the cells, Output for the forget gate. For input gate output, Candidate cell state, For Hadama accumulation, The time decay coefficient, ,in For the Sigmoid function, The amount of data in the previous window. This represents the historical average data volume. The formula, which defines the time interval between the current and previous windows, addresses the lag in predicting burst data peaks in traditional LSTM by introducing a decay term that dynamically varies with the data volume. The initial value of 0.1 was determined by fitting the decay law of 100 sets of burst data samples, and 0.05 was the adjustment coefficient. Through cross-validation optimization, the prediction error was reduced by more than 25%.

[0014] Furthermore, the prediction result correction module of the spatiotemporal prediction engine adopts a feedback iterative correction algorithm, with correction coefficients... The calculation formula is: ,in A window to history The actual amount of data, This represents the amount of historical forecast data. To correct the strength coefficient, The correction effect was determined based on 100 sets of historical error data. The memory decay period is set to 4 hours to ensure that recent errors have a greater impact on the correction. Using the current prediction window as an example, this algorithm reduces long-term prediction errors by 30% through weighted iterative correction of historical prediction errors, making it particularly suitable for scenarios with large fluctuations in data distribution.

[0015] Furthermore, the computing power pre-allocation unit of the resource pre-scheduling center adopts a time-space sensitivity weighted scheduling model, and the computing power allocation amount... The calculation formula is: ,in The baseline computing power is 60% of the node's idle computing power, determined based on the principle of maximizing resource utilization. For the region Spatial sensitivity, ranging from 0 to 1, with values ​​of 0.8 to 1.0 for densely populated areas and 0.2 to 0.5 for suburban areas, is determined through matching with a regional attribute tag library. For time The time sensitivity, ranging from 0 to 1, is 0.7-1.0 during peak hours and 0.3-0.6 during off-peak hours, determined based on historical traffic peak period statistics. For the predicted area In time Data growth rate , , The weights were set to 0.3, 0.3, and 0.4, respectively. The results were determined through comparative experiments on scheduling effects in 10 different scenarios, which reduced the computing power gap in high-sensitivity areas by more than 40%.

[0016] Furthermore, in the dynamic adjustment unit of the cache pool of the resource pre-scheduling center, the cache capacity... The calculation formula is: ,in This represents the maximum cache capacity of the node. To predict the amount of data, This is the largest amount of data in history. This represents the current storage utilization rate of the node. For the region The spatiotemporal correlation with the surrounding area , , The weights are 0.5, 0.3, and 0.2, respectively, and were determined through cache hit rate experiments. This formula improves the cache hit rate of highly correlated regions to over 85%. By integrating predicted data volume, node load, and regional correlation, the formula achieves accurate allocation of cache resources and avoids redundant or insufficient caching.

[0017] Furthermore, in the spatiotemporal cost-optimal path algorithm of the spatiotemporal-aware routing module, the path cost function... The calculation formula is: ,in This refers to the amount of data transferred along the path. For inter-node transmission delay, The value ranges from 0 to 1, representing the spatial correlation between the transmission node and the target region. The cosine similarity is calculated based on the region's location coordinates. This represents the number of hops in the path. Adjacent nodes and The historical collaborative processing success rate, ranging from 0 to 1, is determined by statistical analysis of interaction data from the past 30 days. , , The weighting coefficients are 0.4, 0.3, and 0.3, respectively, and were determined through 1000 path optimization simulations. This reduces the overall transmission cost by 35%. The algorithm achieves global optimization of path cost by integrating transmission volume, latency, spatial correlation, and node coordination.

[0018] Furthermore, the time window alignment unit of the batch-stream collaborative processing engine adopts a spatiotemporal dual-dimensional adaptive window calibration technology. The specific implementation steps are as follows: First, based on the regional data density distribution output by the spatiotemporal prediction engine, the time window is initially divided into a basic segment and an elastic segment. Second, the spatial distribution entropy value of the actual data within the current window is monitored in real time. Through the calculation of the dispersion of regional data points, when the entropy value is higher than the threshold of 0.7, it is determined based on the historical data clustering results and triggers the compression of the elastic segment. Third, the deviation rate between the actual data flow rate and the predicted flow rate is compared synchronously. If the deviation rate exceeds ±20% for three consecutive basic segments, the length of the basic segment is adjusted as a whole. Finally, the time offset of the window edge data is corrected through a timestamp alignment algorithm to ensure that the alignment error of the batch-stream data in the spatiotemporal dimension is less than 500ms. This technology solves the problem of regional data mismatch caused by a single time dimension adjustment through dual-dimensional calibration of spatial distribution and time flow rate. Experimental verification shows that it improves the accuracy of batch-stream collaborative analysis by 28%.

[0019] Furthermore, the result fusion unit of the stream-batch collaborative processing engine adopts spatiotemporal feature anchoring fusion technology. The specific implementation steps are as follows: First, extract the core spatiotemporal feature anchor points of the stream processing results and batch processing results. The stream processing results are anchored to real-time timestamps and regional coordinates, while the batch processing results are anchored to historical time segments and regional cluster centers. Second, align the time axis of the stream and batch processing results using a dynamic time warping algorithm, calculate the matching degree of the time anchor points (value 0-1), with a matching degree ≥ 0.8 indicating strong correlation. Simultaneously, a spatial distance decay model is used to calculate the spatial correlation of regional anchor points. Then... Next, for the result segments corresponding to strongly correlated anchor points, a weighted average fusion is adopted, with the weight of the stream processing result = 0.6 + 0.3 × time matching degree, and the weight of the batch processing result = 1 - the weight of the stream processing result. For the result segments corresponding to weakly correlated anchor points, an anomaly detection mechanism is enabled to prioritize the retention of results with higher data integrity. Finally, through spatiotemporal consistency verification, inconsistent results are corrected a second time. This technology solves the fusion distortion problem caused by spatiotemporal deviations in stream and batch results through precise anchor point matching and hierarchical fusion strategy. Experiments show that the spatiotemporal consistency accuracy of its fusion results is improved by 32%.

[0020] Compared with existing technologies, this efficient big data processing system based on a distributed computing architecture has the following advantages:

[0021] I. This system, through a resource pre-scheduling center, utilizes a spatiotemporal sensitivity weighted scheduling model and a cache pool dynamic adjustment unit to pre-allocate computing, storage, and bandwidth resources accurately. Specifically, the computing power pre-allocation unit dynamically adjusts the computing power allocation based on the spatial sensitivity, temporal sensitivity, and data growth rate of a region, effectively avoiding resource waste and ensuring sufficient resources in high-sensitivity regions during peak hours, thereby significantly improving overall resource utilization. Simultaneously, the cache pool dynamic adjustment unit achieves accurate allocation of cache resources by integrating predicted data volume, node load, and regional correlation, avoiding redundant or insufficient caching and further improving data processing efficiency. Experiments show that this design reduces the computing power gap in high-sensitivity regions by more than 40% and increases the cache hit rate in highly correlated regions to over 85%.

[0022] Second, this system optimizes data transmission paths and accurately fuses stream and batch processing results through a spatiotemporal-aware routing module and a stream-batch collaborative processing engine. The spatiotemporal-aware routing module employs a spatiotemporal cost-optimal path algorithm, comprehensively considering data transmission volume, inter-node transmission latency, spatial correlation, and historical collaborative processing success rate to calculate the lowest-cost transmission path, effectively reducing overall transmission costs. The stream-batch collaborative processing engine, through a time window alignment unit and a result fusion unit, solves the mismatch problem between stream and batch processing in terms of time windows, as well as the result fusion distortion problem caused by spatiotemporal deviations. Specifically, the time window alignment unit adopts spatiotemporal dual-dimensional adaptive window calibration technology to ensure accurate alignment of stream and batch data in the spatiotemporal dimension; the result fusion unit achieves high-precision fusion of stream and batch results through spatiotemporal feature anchoring fusion technology. Experiments show that these designs reduce overall transmission costs by 35%, improve the accuracy of stream and batch data collaborative analysis by 28%, and improve the spatiotemporal consistency accuracy of fusion results by 32%.

[0023] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0025] Figure 1 This is a flowchart illustrating the operation of a high-efficiency big data processing system based on a distributed computing architecture.

[0026] Figure 2 This is an overall flowchart of a high-efficiency big data processing system based on a distributed computing architecture. Detailed Implementation

[0027] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0028] Example 1

[0029] After the spatiotemporal feature extraction module is activated, it collects traffic flow data streams from 200 traffic monitoring points every 30 seconds, including information such as the number of vehicles, vehicle type, and speed. It automatically adds corresponding spatiotemporal tags such as region identifiers and timestamps to each data stream, and then extracts key spatiotemporal feature vectors. The core dynamic spatiotemporal correlation degree is obtained through a dynamic spatiotemporal correlation degree algorithm. Let a certain region... In the time window The internal data stream is ,area With the region The spatial distance is Time window With historical window The time difference is Then the dynamic spatiotemporal correlation degree The calculation formula is: ,in For the region and In time The intersection of data streams within, For the union, , , Weighting coefficients; The spatial reference distance Using the time reference window as a reference, this step aims to provide accurate feature support for subsequent traffic flow prediction by comprehensively analyzing the intersection and union of traffic flow data between different monitoring points, the spatial distance between regions, and the time difference between the current time window and the historical window. This avoids the feature bias caused by considering only the time or spatial dimension and ensures that the subsequent prediction results can fully reflect the spatiotemporal correlation of traffic flow.

[0030] After receiving the spatiotemporal feature vectors, the spatiotemporal prediction engine first processes the data using the dynamic decaying LSTM algorithm in the time series prediction submodule. The formula is as follows: ,in Current time The state of the cells, Output for the forget gate. For input gate output, Candidate cell state, For Hadama accumulation, This is the time decay coefficient; The algorithm uses the time interval between the current window and the previous window as a reference. It incorporates historical traffic flow patterns and uses a time decay coefficient to reflect the weight of different historical data on the current prediction, initially predicting the spatiotemporal distribution of traffic flow in various areas over the next 15 minutes. Next, the spatial correlation submodule analyzes the traffic flow relationships between adjacent monitoring points to further optimize the spatial rationality of the prediction results. Finally, the prediction result correction module uses a feedback iterative correction algorithm, with the formula: ,in A window to history The actual amount of data, This represents the amount of historical forecast data. To correct the strength coefficient, For memory decay period, Using the current prediction window as a reference, and referring to the deviation between the historical actual traffic flow data and the corresponding prediction data in 15-minute windows over the past 3 hours, a correction coefficient is calculated to dynamically adjust the current prediction result. Finally, an accurate traffic flow spatiotemporal distribution prediction table is output. The purpose of this step is to effectively reduce prediction deviations caused by unforeseen circumstances and improve the accuracy of traffic flow distribution prediction through a three-layer processing logic of time-series prediction, spatial correlation, and iterative correction. This provides a reliable decision-making basis for subsequent resource scheduling and traffic scheduling, and avoids resource waste or improper traffic scheduling caused by inaccurate predictions.

[0031] After obtaining the prediction table, the resource pre-scheduling center first collects the resource status of each edge computing node through a real-time monitoring system, and then pre-allocates computing, storage, and bandwidth resources based on this information. The computing power pre-allocation unit adopts a time-space sensitivity weighted scheduling model, with the following formula: ,in As a benchmark computing power, For the region Spatial sensitivity, For time Time sensitivity For the predicted area In time Data growth rate , , As weights, for spatially sensitive areas such as around transportation hubs, time-sensitive periods such as morning and evening rush hours, and monitoring points with high predicted traffic flow growth rates, priority is given to increasing the computing power allocation of corresponding computing nodes; the cache pool dynamic adjustment unit dynamically adjusts the size of each node's cache pool based on the predicted data volume, the current storage utilization rate of the nodes, and the spatiotemporal correlation between regions, through cache capacity calculation logic, prioritizing the cache space for frequently accessed data, as shown in the formula: ,in This represents the maximum cache capacity of the node. To predict the amount of data, This is the largest amount of data in history. This represents the current storage utilization rate of the node. For the region The spatiotemporal correlation with the surrounding area , , The purpose of this step is to achieve on-demand pre-allocation of resources, avoiding processing delays caused by insufficient resources in the traditional post-allocation model. At the same time, through differentiated computing power and cache allocation strategies, it improves resource utilization, reduces idle resources, and ensures that each node can efficiently process traffic data in its corresponding area, supporting the achievement of the overall data processing delay target.

[0032] The spatiotemporal awareness routing module calculates the data transmission path cost based on the predicted spatiotemporal distribution of traffic flow using a spatiotemporal cost-optimal path algorithm. The formula is as follows: ,in This refers to the amount of data transferred along the path. For inter-node transmission delay, To determine the spatial correlation between the transmission node and the target area, This represents the number of hops in the path. Adjacent nodes and The historical success rate of collaborative processing , , Using weighted coefficients, this algorithm comprehensively considers multiple factors such as data transmission volume along the path, transmission delay between nodes, spatial correlation between transmission nodes and target areas, path hop count, and historical collaborative processing success rate of adjacent nodes. It selects the transmission path with the lowest cost to accurately transmit traffic flow data from each monitoring point to the corresponding computing node. The purpose of this step is to reduce data delay and loss during transmission by optimizing the data transmission path, avoiding data arrival delays caused by unreasonable transmission paths, ensuring that computing nodes can obtain traffic flow data for processing in a timely manner, reducing data retransmission rate, reducing bandwidth resource waste, and further ensuring the real-time performance of overall data processing.

[0033] After the batch processing engine is started, the time window alignment unit first uses the spatiotemporal dual-dimensional adaptive window calibration technology of claim 8. First, based on the predicted regional traffic flow data density distribution, the time window is divided into a basic segment and an elastic segment. Then, the spatial distribution entropy value of the actual traffic flow data in the current window is monitored in real time. When the entropy value is higher than a set threshold, the elastic segment is compressed to improve the timeliness of data processing. Then, the deviation rate between the actual traffic flow rate and the predicted flow rate is compared synchronously. If the deviation rate exceeds ±20% for three consecutive basic segments, the length of the basic segment is adjusted as a whole to adapt to the actual data changes. Finally, the time offset of the window edge data is corrected by the timestamp alignment algorithm to achieve the unification of the time window for stream processing and batch processing. The purpose of this step is to solve the data processing discontinuity problem caused by the inconsistency of the time window in traditional stream and batch processing. Adaptive window calibration ensures the consistency of stream and batch data in the time dimension, provides a unified time reference for subsequent result fusion, and improves the adaptability of data processing to actual traffic flow changes, avoiding processing lag or data redundancy caused by fixed windows.

[0034] The result fusion unit employs spatiotemporal feature anchoring fusion technology. First, it extracts the core spatiotemporal feature anchors from the streaming and batch processing results. Next, it aligns the time axes of the streaming and batch results using a dynamic time warping algorithm, calculating the matching degree of the time anchors. Simultaneously, it uses a spatial distance attenuation model to calculate the spatial correlation of regional anchors. Then, for result segments corresponding to strongly correlated anchors, a weighted average fusion is applied; for result segments corresponding to weakly correlated anchors, an anomaly detection mechanism is enabled, prioritizing the retention of results with higher data integrity. Finally, a spatiotemporal consistency check is performed to correct inconsistent results, ultimately outputting complete and accurate traffic flow monitoring and analysis results, which are pushed to the traffic signal dispatching system in real time. This step effectively combines the real-time nature of streaming processing with the accuracy of batch processing through feature anchoring, multi-dimensional fusion, and consistency check, avoiding the limitations of a single processing mode. It outputs high-quality traffic flow analysis results, providing precise support for the traffic signal dispatching system, effectively alleviating traffic congestion, and improving road traffic efficiency.

[0035] Example 2

[0036] After the spatiotemporal feature extraction module is activated, it collects parcel data streams uploaded every minute from 500 delivery stations, adds spatiotemporal tags such as station number and sending / receiving timestamps to each data stream, and then extracts dynamic spatiotemporal correlation. This correlation is calculated by the dynamic spatiotemporal correlation algorithm in claim 2, taking into account factors such as the intersection and union of parcel data between different stations, the geographical distance between stations, and the time difference between the current time window and the historical time window.

[0037] After receiving the spatiotemporal feature vector, the spatiotemporal prediction engine starts the time-series prediction submodule, which uses the dynamic decay LSTM algorithm to analyze historical parcel delivery and receipt time-series data. Combined with influencing factors such as holidays and promotional activities, it predicts the spatiotemporal distribution of parcels at each station in the next 2 hours. The spatial correlation submodule analyzes the parcel correlation relationships between adjacent delivery areas and stations on the same delivery route to help optimize the prediction results. The prediction result correction module uses a feedback iterative correction algorithm, which refers to the deviation between the historical actual parcel volume and the predicted parcel volume in the same period of the past 7 days, calculates the correction coefficient, iteratively corrects the current prediction results, and outputs an accurate parcel spatiotemporal distribution prediction table.

[0038] After obtaining the forecast table, the resource pre-scheduling center, in conjunction with the resource status of each node in the distributed computing cluster, pre-allocates computing, storage, and bandwidth resources. The computing power pre-allocation unit adopts a time-spatial sensitivity weighted scheduling model, increasing the computing power allocation of corresponding computing nodes for sites with high spatial sensitivity, such as e-commerce concentrated areas, time-sensitive periods, such as promotional activities, and areas with high predicted parcel growth rates. The cache pool dynamic adjustment unit dynamically adjusts the cache pool size based on the predicted parcel data volume, the current storage utilization rate of nodes, and the time-spatial correlation between sites, through cache capacity-related calculation logic, to ensure efficient caching and retrieval of parcel logistics information.

[0039] Based on the predicted spatiotemporal distribution of packages, the spatiotemporal-aware routing module uses a spatiotemporal cost-optimal path algorithm to calculate the data transmission path. It comprehensively weighs factors such as the amount of package data transmitted along the path, the transmission delay between nodes, the spatial correlation between the transmission node and the target delivery area, the number of path hops, and the historical collaborative processing success rate of adjacent nodes to select the optimal transmission path and efficiently transmit the package data of each station to the target computing node.

[0040] After the stream-batch collaborative processing engine starts, the time window alignment unit adopts a spatiotemporal dual-dimensional adaptive window calibration technology. First, based on the predicted regional package data density distribution, the time window is divided into a basic segment and an elastic segment. The spatial distribution entropy value of the actual package data within the current window is monitored in real time. When the entropy value is higher than the threshold, elastic segment compression is triggered. The deviation rate between the actual package data flow rate and the predicted flow rate is compared synchronously. If the deviation rate exceeds ±20% for three consecutive basic segments, the length of the basic segment is adjusted as a whole. The time offset of the window edge data is corrected through a timestamp alignment algorithm to achieve the unification of the time window for stream processing and batch processing.

[0041] The result fusion unit employs spatiotemporal feature anchoring fusion technology to extract core spatiotemporal feature anchors from stream processing and batch processing results. It aligns the timelines of stream and batch results using a dynamic time warping algorithm, calculates the time anchor matching degree, and simultaneously uses a spatial distance decay model to calculate the spatial correlation of regional anchors. Weighted average fusion is applied to result segments corresponding to strongly correlated anchors, while an anomaly detection mechanism is enabled for result segments corresponding to weakly correlated anchors, prioritizing the retention of results with higher data integrity. Finally, spatiotemporal consistency verification is used to perform secondary corrections on inconsistent results, outputting the final package data processing results. This provides real-time and accurate data support for the logistics delivery route planning system, optimizing delivery routes and improving delivery efficiency.

[0042] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A high-efficiency big data processing system based on a distributed computing architecture, characterized in that, The system comprises the following components: a spatiotemporal feature extraction module, a spatiotemporal prediction engine, a resource pre-scheduling center, a spatiotemporal-aware routing module, and a stream-batch collaborative processing engine; The spatiotemporal feature extraction module: collects spatiotemporal labels of the data stream and extracts spatiotemporal feature vectors, wherein the spatiotemporal feature vectors include dynamic spatiotemporal correlation degree; The spatiotemporal prediction engine predicts the spatiotemporal distribution of future data streams based on the spatiotemporal feature vectors and outputs a prediction table. The spatiotemporal prediction engine includes a time series prediction submodule, a spatial correlation submodule, and a prediction result correction module. The resource pre-scheduling center: allocates computing, storage and bandwidth resources in advance according to the prediction table and node resource status. The resource pre-scheduling center includes a computing power pre-allocation unit and a cache pool dynamic adjustment unit. The spatiotemporal-aware routing module optimizes the data transmission path based on the predicted spatiotemporal distribution and calculates the path cost using a spatiotemporal cost-optimal path algorithm. The stream-batch collaborative processing engine unifies the time windows for stream processing and batch processing to achieve collaborative data processing. The stream-batch collaborative processing engine includes a time window alignment unit and a result fusion unit. In the spatiotemporal cost-optimal path algorithm of the spatiotemporal-aware routing module, the path cost function The calculation formula is: ,in This refers to the amount of data transferred along the path. For inter-node transmission delay, To determine the spatial correlation between the transmission node and the target area, This represents the number of hops in the path. Adjacent nodes and The historical success rate of collaborative processing , , These are the weighting coefficients; The time window alignment unit of the batch processing engine adopts a spatiotemporal dual-dimensional adaptive window calibration technology. The specific implementation steps are as follows: First, based on the regional data density distribution output by the spatiotemporal prediction engine, the time window is initially divided into a basic segment and an elastic segment; second, the spatial distribution entropy value of the actual data within the current window is monitored in real time. When the entropy value is higher than the threshold, elastic segment compression is triggered; third, the deviation rate between the actual data flow rate and the predicted flow rate is compared synchronously. If the deviation rate exceeds ±20% for three consecutive basic segments, the length of the basic segment is adjusted as a whole; finally, the time offset of the window edge data is corrected through a timestamp alignment algorithm. The result fusion unit of the stream-batch collaborative processing engine adopts spatiotemporal feature anchoring fusion technology. The specific implementation steps are as follows: First, extract the core spatiotemporal feature anchor points of the stream processing results and batch processing results; second, align the time axis of the stream and batch results using a dynamic time warping algorithm, calculate the matching degree of the time anchor points, and simultaneously use a spatial distance decay model to calculate the spatial correlation of the regional anchor points; third, for the result segments corresponding to strongly correlated anchor points, use a weighted average fusion; for the result segments corresponding to weakly correlated anchor points, enable an anomaly detection mechanism to prioritize retaining results with higher data integrity; finally, perform a secondary correction on inconsistent results through spatiotemporal consistency verification.

2. The high-efficiency big data processing system based on a distributed computing architecture according to claim 1, characterized in that, The dynamic spatiotemporal correlation degree extracted by the spatiotemporal feature extraction module is obtained through the following algorithm: Let a certain region In the time window The internal data stream is ,area With the region The spatial distance is Time window With historical window The time difference is Then the dynamic spatiotemporal correlation degree The calculation formula is: ,in For the region and In time The intersection of data streams within, For the union, , , Weighting coefficients; The spatial reference distance This serves as the time reference window.

3. The high-efficiency big data processing system based on a distributed computing architecture according to claim 1, characterized in that, The temporal prediction submodule of the spatiotemporal prediction engine adopts the dynamic decay LSTM algorithm, and its cell state update formula is as follows: ,in Current time The state of the cells, Output for the forget gate. For input gate output, Candidate cell state, For Hadama accumulation, This is the time decay coefficient; This represents the time interval between the current window and the previous window.

4. The efficient big data processing system based on a distributed computing architecture according to claim 1, characterized in that, The prediction result correction module of the spatiotemporal prediction engine adopts a feedback iterative correction algorithm, with correction coefficients... The calculation formula is: ,in A window to history The actual amount of data, This represents the amount of historical forecast data. To correct the strength coefficient, For memory decay period, This is the current forecast window.

5. The high-efficiency big data processing system based on a distributed computing architecture according to claim 1, characterized in that, The computing power pre-allocation unit of the resource pre-scheduling center adopts a time-space sensitivity weighted scheduling model, with a computing power allocation amount... The calculation formula is: ,in As a benchmark computing power, For the region Spatial sensitivity, For time Time sensitivity For the predicted area In time Data growth rate , , For weights.

6. The efficient big data processing system based on a distributed computing architecture according to claim 1, characterized in that, In the dynamic adjustment unit of the cache pool of the resource pre-scheduling center, the cache capacity The calculation formula is: ,in This represents the maximum cache capacity of the node. To predict the amount of data, This is the largest amount of data in history. This represents the current storage utilization rate of the node. For the region The spatiotemporal correlation with the surrounding area , , For weights.

Images