A distributed data processing method and system thereof

Abstract

The present invention provides a distributed data processing method and a system thereof, which relate to the field of data processing technology, and include collecting an initial data set to be processed from a target area, pre-processing the initial data set to be processed in a distributed manner to obtain a first data set; dynamically and recursively partitioning the target area through a spatial partition tree SPT, and allocating data points in the first data set to each of the divided sub-areas and marking them to obtain a second data set with a spatial partition identifier; receiving a real-time data stream, and dynamically integrating the real-time data stream into the second data set according to spatial coordinates to obtain a third data set; locating the position of the data point in the spatial partition tree SPT based on the spatial partition identifier, and allocating the data points in the third data set in the same sub-area to the same data processing node; executing data processing tasks in parallel on each data processing node, and then aggregating the obtained initial results to form a final result.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a distributed data processing method and system thereof. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, distributed data processing technology has gradually become an important means of processing massive data and improving data processing efficiency. Especially in the fields of geographic information systems (GIS), the Internet of Things (IoT), smart cities, etc., the data is not only large in volume but also usually has spatial attributes, that is, data points are associated with specific geographical locations. The processing and analysis of such data puts forward higher requirements for real-time, accuracy and scalability. Traditional data processing methods often face challenges such as slow processing speed, high resource consumption, and poor real-time performance when facing large-scale spatial data.

[0003] Therefore, it is necessary to provide a distributed data processing method and system to solve the above technical problems. Summary of the invention

[0004] To solve the above technical problems, the present invention provides a distributed data processing method and system thereof, which combines the advantages of the distributed computing framework Spark and spatial partitioning technology, and realizes efficient and real-time processing of large-scale spatial data through steps such as dynamic recursive division of target areas, real-time integration into real-time data streams, data point positioning and node allocation based on spatial partition identifiers, and parallel execution of data processing tasks and result aggregation.

[0005] The present invention provides a distributed data processing method, which comprises the following steps:

[0006] Collecting an initial data set to be processed from a target area, and using a distributed computing framework Spark to perform distributed preprocessing on the initial data set to be processed to obtain a first data set, wherein the first data set includes a plurality of data points;

[0007] Dynamically recursively partitioning the target area through a spatial partition tree SPT, and allocating data points in the first data set to each of the divided sub-areas, while adding a spatial partition identifier to obtain a second data set with the spatial partition identifier;

[0008] receiving a real-time data stream, and dynamically integrating the real-time data stream into the second data set according to spatial coordinates to obtain a third data set;

[0009] Positioning the data points in the spatial partition tree SPT based on the spatial partition identifier, and allocating the data points in the third data set in the same sub-area to the same data processing node according to the positioned positions;

[0010] On each data processing node, data processing tasks are executed in parallel through the distributed computing framework Spark, and then the initial results are aggregated to form the final results.

[0011] Preferably, the collecting of the to-be-processed initial data set from the target area and the use of the distributed computing framework Spark to perform distributed preprocessing on the to-be-processed initial data set to obtain the first data set includes:

[0012] Collecting an initial data set to be processed from the target area by a data acquisition component, wherein the initial data set to be processed includes multiple types of data related to the target area;

[0013] In the environment of the distributed computing framework Spark, the initial data set to be processed is divided into multiple data blocks, and a preprocessing operation is performed on multiple computing nodes of Spark to obtain a preprocessed first data set.

[0014] Preferably, the target area is dynamically and recursively divided by a spatial partition tree SPT, and data points in the first data set are allocated to each of the divided sub-areas, and a spatial partition identifier is added to obtain a second data set with a spatial partition identifier, including:

[0015] Constructing a spatial partition tree SPT, and subdividing the target area into finer-grained sub-areas through the constructed spatial partition tree SPT according to a preset division rule until a division stop condition is met;

[0016] Traversing each data point in the first data set, and assigning each data point one by one to a sub-region located in the spatial partition tree SPT according to the spatial coordinates of each data point;

[0017] During allocation, a spatial partition identifier is added to each data point allocated to a sub-region, wherein the spatial partition identifier uniquely and clearly indicates the sub-region to which the data point belongs, and is associated with a leaf node in the spatial partition tree SPT to obtain a second data set with the spatial partition identifier.

[0018] Preferably, the receiving of the real-time data stream and dynamically integrating the real-time data stream into the second data set according to the spatial coordinates to obtain the third data set includes:

[0019] capturing a real-time data stream associated with a target area in real time, wherein the real-time data stream contains dynamically changing spatial coordinates;

[0020] Preprocessing the captured real-time data stream to ensure that the real-time data stream is consistent with the data format in the second data set and the spatial coordinates are accurate;

[0021] According to the spatial coordinates of each data point in the preprocessed real-time data stream, the sub-region to which it belongs is located in the spatial partition tree SPT;

[0022] Each data point of the located real-time data stream is dynamically inserted into the data point set of the corresponding sub-area in the second data set, and the data record of the associated leaf node in the spatial partition tree SPT is updated to obtain a third data set of the fused real-time data stream.

[0023] Preferably, the positioning of the data point in the space partition tree SPT based on the space partition identifier, and allocating the data points in the third data set in the same sub-area to the same data processing node according to the positioned position, includes:

[0024] Traverse each data point in the third data set and read the spatial partition identifier attached thereto;

[0025] Using the read spatial partition identifier, locate the sub-region to which the data point belongs in the spatial partition tree SPT, and determine the set of data points in the same sub-region;

[0026] According to the pre-constructed data allocation mapping table, the set of data points in the same sub-area is allocated as a whole to the corresponding data processing node to ensure that the data points in the same sub-area are processed on the same node, wherein the data allocation mapping table records the mapping relationship between each sub-area and the corresponding data processing node.

[0027] Preferably, the data processing tasks are executed in parallel on each data processing node through the distributed computing framework Spark, and then the obtained initial results are aggregated to form the final results, including:

[0028] On each data processing node that is assigned a data point, start the execution environment of the distributed computing framework Spark;

[0029] Write and deploy corresponding Spark jobs according to the requirements of data processing tasks, where each Spark job is designed to process the set of data points assigned to the node;

[0030] Execute multiple Spark jobs in parallel and collect the initial results of the output after each Spark job is completed;

[0031] The initial results from different data processing nodes are merged and integrated through Spark's aggregation mechanism to form a globally consistent final result.

[0032] The present invention also provides a distributed data processing system for executing a distributed data processing method, the processing system comprising:

[0033] A data preprocessing module is used to collect an initial data set to be processed from a target area, and to perform distributed preprocessing on the initial data set to be processed using a distributed computing framework Spark to obtain a first data set, wherein the first data set includes a plurality of data points;

[0034] A data allocation module, used for dynamically and recursively partitioning the target area through a spatial partition tree SPT, and allocating data points in the first data set to each of the divided sub-areas, while adding a spatial partition identifier to obtain a second data set with the spatial partition identifier;

[0035] A real-time data fusion module, used for receiving a real-time data stream, and dynamically merging the real-time data stream into the second data set according to spatial coordinates to obtain a third data set;

[0036] a node allocation module, configured to locate the position of the data point in the space partition tree SPT based on the space partition identifier, and allocate the data points in the third data set in the same sub-area to the same data processing node according to the located position;

[0037] The result aggregation module is used to execute data processing tasks in parallel on each data processing node through the distributed computing framework Spark, and then aggregate the initial results to form the final results.

[0038] Compared with the related art, the distributed data processing method and system provided by the present invention have the following beneficial effects:

[0039] The present invention first collects the initial data set to be processed from the target area and uses the distributed computing framework Spark for distributed preprocessing. This step aims to convert the raw data into a format suitable for subsequent processing and analysis, while using Spark's distributed computing capabilities to improve processing efficiency.

[0040] Secondly, the target area is dynamically and recursively divided through the spatial partition tree SPT, and the preprocessed data points are allocated to each sub-area after division. The spatial partition tree SPT can flexibly divide the data according to the spatial distribution characteristics of the data to ensure that the data points are reasonably allocated to each sub-area. At the same time, a spatial partition identifier is added to each data point to facilitate subsequent data positioning and node allocation.

[0041] Next, the real-time data stream is received and dynamically integrated into the divided data set according to the spatial coordinates. This step enables the method of the present invention to process dynamically changing spatial data, thereby improving the real-time performance and accuracy of data processing.

[0042] Then, the data points are located in the spatial partition tree SPT based on the spatial partition identifier, and the data points in the same sub-area are assigned to the same data processing node according to the positioning result. This step aims to reduce the data transmission overhead between nodes and improve the locality and efficiency of data processing.

[0043] Finally, on each data processing node, the distributed computing framework Spark is used to execute data processing tasks in parallel, and the initial results are aggregated to form the final results. This step fully utilizes Spark's parallel computing capabilities and aggregation mechanism to improve the speed and accuracy of data processing.

[0044] In summary, the present invention proposes an innovative distributed data processing method, which combines the advantages of spatial partitioning technology and the distributed computing framework Spark to achieve effective processing and analysis of large-scale spatial data. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flow chart of a distributed data processing method provided by the present invention;

[0046] Figure 2 A module structure diagram of a distributed data processing system provided by the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0048] It should also be noted that, for ease of description, only the parts related to the present invention, but not all of the contents, are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0049] Embodiment 1

[0050] The present invention provides a distributed data processing method, referring to Figure 1As shown, the processing method comprises the following steps:

[0051] S1: Collecting an initial data set to be processed from a target area, and using a distributed computing framework Spark to perform distributed preprocessing on the initial data set to be processed to obtain a first data set, wherein the first data set includes a plurality of data points.

[0052] Specifically, step S1 includes the following steps:

[0053] S11: collecting an initial data set to be processed from the target area through a data acquisition component, wherein the initial data set to be processed includes multiple types of data related to the target area.

[0054] In this embodiment, the data acquisition component needs to obtain raw data from various data sources, which can be but not limited to sensor networks, Internet of Things devices, and database systems. The quality of data acquisition directly affects the effect and accuracy of subsequent data processing. Therefore, designing an efficient and reliable data acquisition mechanism is crucial for distributed data processing systems.

[0055] In the present invention, step S11 collects the initial data set to be processed from the target area through the data acquisition component, wherein the initial data set to be processed includes multiple types of data related to the target area. The specific implementation process is as follows:

[0056] The data acquisition component is connected to various data sources through the network, such as sensor networks, IoT devices, database systems, etc.

[0057] Data is collected from these data sources periodically or in real time. For example, a sensor network can send environmental monitoring data every minute, IoT devices can upload user behavior data in real time, and a database system can export historical records.

[0058] The collected data is transmitted to the data processing system through network transmission protocols (such as HTTP, MQTT). In order to ensure the reliability and security of data transmission, data compression, encryption and other technologies can also be used.

[0059] The received data is temporarily stored in a buffer or temporary storage, awaiting further processing.

[0060] S12: In the environment of the distributed computing framework Spark, the initial data set to be processed is divided into multiple data blocks, and a preprocessing operation is performed on multiple computing nodes of Spark to obtain a preprocessed first data set.

[0061] In this embodiment, data preprocessing is an important part of data processing, especially in a distributed computing environment. Raw data usually contains noise, missing values, and inconsistent information, and needs to be preprocessed before it can be used for subsequent analysis and processing. The purpose of preprocessing is to improve data quality, ensure data consistency and accuracy, and thus improve the efficiency of subsequent processing and the reliability of results.

[0062] In the present invention, the specific implementation process of step S12 is as follows:

[0063] Data segmentation: Split the collected initial data set into multiple data blocks. The Spark framework will automatically split the data set into multiple partitions, each partition corresponding to a data block.

[0064] Task allocation: Allocate each data block to different computing nodes in the Spark cluster. Spark's scheduler dynamically allocates tasks based on the node load and resource availability.

[0065] Preprocessing operations: Preprocessing operations are performed on each computing node, including but not limited to:

[0066] Data cleaning: Remove missing values, outliers, and duplicate records.

[0067] Formatting: Convert data into a unified format to ensure consistency in subsequent processing.

[0068] Filter: Filter out data points that do not meet the requirements based on preset conditions.

[0069] Result summary: The preprocessed data blocks are reassembled into the first dataset for use in subsequent steps.

[0070] S2: dynamically and recursively partition the target area through a spatial partition tree SPT, and distribute the data points in the first data set to each of the divided sub-areas, while adding a spatial partition identifier to obtain a second data set with a spatial partition identifier.

[0071] Specifically, step S2 includes the following steps:

[0072] S21: constructing a space partition tree SPT, and subdividing the target area into sub-areas with smaller granularity through the constructed space partition tree SPT according to a preset division rule, until a division stop condition is met.

[0073] In distributed data processing, spatial partitioning is one of the key steps to improve data processing efficiency. Although the traditional grid partitioning method is simple, it may cause uneven data distribution when processing large-scale data, affecting processing performance. As a hierarchical data structure, the spatial partition tree (SPT) can dynamically partition data according to its distribution, ensuring that the amount of data in each partition is relatively balanced. Therefore, building an SPT is crucial to optimizing the data processing process.

[0074] In the present invention, the specific implementation process of step S21 is as follows:

[0075] Initial region: Take the entire target region as a root node.

[0076] Recursive partitioning: For each node, if the number of data points it contains exceeds a predetermined threshold, it is further divided into target areas based on the number of data points.

[0077] Termination condition: When the number of data points in a sub-region is less than or equal to the predetermined threshold, the division stops and the sub-region becomes a leaf node.

[0078] For example, assume that there is a target area containing 1000 data points, and the preset data point quantity threshold is 100 data points.

[0079] The entire target area contains 1000 data points.

[0080] Since the number of data points in the entire area exceeds 100, the entire area is divided into four sub-areas.

[0081] The number of data points in each sub-region are: 250, 250, 250, 250 respectively.

[0082] The number of data points in each of the four sub-regions exceeds 100, so each sub-region needs to be further divided.

[0083] Each sub-region is further divided into four smaller sub-regions.

[0084] The number of data points in each small sub-area is 62.5, 62.5, 62.5, 62.5 respectively (in actual operation, the number of data points will be an integer, but floating point numbers are used here to simplify the explanation).

[0085] When the number of data points in a sub-region does not exceed 100, stop further division.

[0086] In this example, after two divisions, the number of data points in each minimum sub-region is 62 or 63, which meets the termination condition.

[0087] The final space partition tree (SPT) consists of:

[0088] Root node: the entire target area, containing 1000 data points.

[0089] First-level leaf node: four sub-regions, each sub-region contains 250 data points.

[0090] Second-level leaf nodes: sixteen smallest sub-regions, each of which contains 62 or 63 data points.

[0091] S22: traverse each data point in the first data set, and assign each data point one by one to a sub-area located in the space partition tree SPT according to the spatial coordinates of each data point.

[0092] In this embodiment, after the spatial partition tree SPT is constructed, the data points in the first data set need to be allocated to each sub-region. This step ensures that each data point can be accurately located in the sub-region to which it belongs, providing a basis for subsequent data processing.

[0093] In the present invention, the specific implementation process of step S22 is as follows:

[0094] Data point traversal: traverse each data point in the first data set and read its spatial coordinates.

[0095] Sub-region location: According to the spatial coordinates of the data point, find the sub-region to which it belongs in the SPT. This can be done by starting from the root node and searching down layer by layer until a suitable leaf node is found.

[0096] Data point allocation: Assign each data point to the found sub-area and record the information of the data point.

[0097] S23: When allocating, add a spatial partition identifier to each data point allocated to the sub-area, wherein the spatial partition identifier uniquely and clearly indicates the sub-area to which the data point belongs, and establishes an association with a leaf node in the spatial partition tree SPT to obtain a second data set with the spatial partition identifier.

[0098] In this embodiment, after the data point allocation is completed, a spatial partition identifier needs to be added to each data point allocated to the sub-region. The spatial partition identifier is used to uniquely and clearly indicate the sub-region to which the data point belongs, and is associated with the leaf node in the SPT. This step ensures that the data point can be quickly located and accessed in subsequent data processing.

[0099] In the present invention, the specific implementation process of step S23 is as follows:

[0100] Logo generation: Generate a unique spatial partition logo for each sub-area to ensure the uniqueness and clarity of the logo.

[0101] Adding identifiers: When allocating data points, add corresponding spatial partition identifiers for each data point.

[0102] Data record update: Update the data record of the corresponding leaf node in the SPT, and record the data points in the sub-area and their spatial partition identifiers.

[0103] Dataset generation: The data points with spatial partition identifiers are reassembled into a second dataset for use in subsequent steps.

[0104] S3: receiving a real-time data stream, and dynamically integrating the real-time data stream into the second data set according to spatial coordinates to obtain a third data set.

[0105] Specifically, step S3 includes the following steps:

[0106] S31: Capturing a real-time data stream related to the target area in real time, wherein the real-time data stream contains dynamically changing spatial coordinates.

[0107] In distributed data processing systems, the processing of real-time data streams is one of the key links. Real-time data streams usually contain dynamically changing spatial coordinates and other related information, which need to be captured and integrated into existing data sets in a timely manner to maintain the timeliness and accuracy of the data. Therefore, designing an efficient real-time data capture mechanism is crucial for distributed data processing systems.

[0108] In the present invention, the specific implementation process of step S31 is as follows:

[0109] Real-time data streams can also come from a variety of data sources, such as sensor networks, IoT devices, real-time databases, etc. Data acquisition components are also used to connect to these data sources through the network.

[0110] Capture data streams from these data sources in real time. For example, sensor networks can send environmental monitoring data every second, IoT devices can upload user behavior data in real time, and real-time databases can provide the latest data records.

[0111] The captured real-time data is transmitted to the data processing system through network transmission protocols (such as HTTP, MQTT). In order to ensure the real-time and reliability of data transmission, data compression, encryption and other technologies can be used.

[0112] S32: Preprocess the captured real-time data stream to ensure that the real-time data stream is consistent with the data format in the second data set and the spatial coordinates are accurate.

[0113] In this embodiment, after the real-time data stream is captured, it needs to be preprocessed to ensure that the real-time data stream is consistent with the data format in the existing data set and the spatial coordinates are accurate. The purpose of preprocessing is to improve data quality, ensure data consistency and accuracy, thereby improving the efficiency of subsequent processing and the reliability of results.

[0114] In the present invention, the specific implementation process of step S32 is as follows:

[0115] Data cleaning: Remove missing values, outliers, and duplicate records from real-time data streams.

[0116] Formatting: Convert data in real-time data streams into a format consistent with existing data sets to ensure consistency in subsequent processing.

[0117] Coordinate verification: Verify that the spatial coordinates of each data point in the real-time data stream are accurate and make corrections if necessary.

[0118] Data annotation: Add necessary metadata to the preprocessed data points, such as timestamp, data source identifier, etc.

[0119] S33: According to the spatial coordinates of each data point in the preprocessed real-time data stream, the sub-region to which it belongs is located in the spatial partition tree SPT.

[0120] In this embodiment, in the preprocessed real-time data stream, it is necessary to locate the sub-region to which each data point belongs in the spatial partition tree (SPT) according to the spatial coordinates of each data point. This step ensures that each real-time data point can be accurately assigned to the corresponding sub-region, providing a basis for subsequent data fusion.

[0121] In the present invention, the specific implementation process of step S33 is as follows:

[0122] Data point traversal: Traverse each data point in the preprocessed real-time data stream and read its spatial coordinates.

[0123] Sub-region location: According to the spatial coordinates of the data point, find the sub-region to which it belongs in the SPT. This can be done by starting from the root node and searching down layer by layer until a suitable leaf node is found.

[0124] Data point recording: Record each data point and the sub-region information to which it belongs for subsequent data fusion.

[0125] S34: Dynamically insert each data point of the located real-time data stream into the data point set of the corresponding sub-area in the second data set, and simultaneously update the data records of the associated leaf nodes in the spatial partition tree SPT to obtain a third data set of the fused real-time data stream.

[0126] In this embodiment, after locating the sub-region of each data point in the real-time data stream, it is necessary to dynamically insert these data points into the data point set of the corresponding sub-region in the second data set, and update the data record of the associated leaf node in the SPT. This step ensures that the real-time data stream can be integrated into the existing data set in a timely manner, maintaining the timeliness and integrity of the data.

[0127] In the present invention, the specific implementation process of step S34 is as follows:

[0128] Data point insertion: insert each data point into the set of data points of the corresponding sub-region in the second data set.

[0129] Data record update: Update the data record of the corresponding leaf node in the SPT, recording the data points and related information in the sub-area.

[0130] Dataset generation: The data points fused with the real-time data stream are reassembled into a third dataset for use in subsequent steps.

[0131] S4: Locate the position of the data point in the space partition tree SPT based on the space partition identifier, and distribute the data points in the third data set in the same sub-area to the same data processing node according to the located position.

[0132] Specifically, step S4 includes the following steps:

[0133] S41: traverse each data point in the third data set and read the spatial partition identifier attached thereto.

[0134] In this embodiment, efficient management and allocation of data points is the key to ensuring data processing performance and accuracy. In particular, when processing large-scale data sets, rationally allocating data points to different data processing nodes can significantly improve data processing efficiency. To this end, the present invention proposes a method based on spatial partition identification to locate the position of data points in a spatial partition tree (SPT) and allocate data points in the same sub-area to the same data processing node.

[0135] In the present invention, the specific implementation process of step S41 is as follows:

[0136] Data point traversal: Traverse each data point in the third dataset and read its attached spatial partition identifier. This step ensures that each data point can be accurately identified and processed.

[0137] Identification reading: Read the spatial partition identification of each data point. These identifications are assigned to each data point in the previous stage (S23) and are used to uniquely and clearly indicate the sub-region to which the data point belongs.

[0138] Data recording: The read spatial partition identifier is recorded together with the data point information to prepare for the subsequent data point allocation.

[0139] S42: Using the read spatial partition identifier, locate the sub-region to which the data point belongs in the spatial partition tree SPT, and determine the set of data points in the same sub-region.

[0140] In this embodiment, after reading the spatial partition identifier of each data point, it is necessary to use these identifiers to locate the sub-region to which the data point belongs in the spatial partition tree (SPT) and determine the set of data points in the same sub-region. This step ensures that the data points can be accurately assigned to the corresponding sub-regions, providing a basis for subsequent data processing.

[0141] In the present invention, the specific implementation process of step S42 is as follows:

[0142] Identity resolution: Resolve the spatial partition identity of each data point to determine its corresponding sub-region. This can be achieved by querying the index structure of the SPT, which records the identity of each sub-region and its corresponding leaf node.

[0143] Sub-region positioning: According to the parsed sub-region identifier, find the corresponding leaf node in the SPT to determine the sub-region to which the data point belongs.

[0144] Data point set: Data points belonging to the same sub-area are classified into the same data point set. This step ensures that all data points belonging to the same sub-area can be processed centrally, improving the efficiency and consistency of data processing.

[0145] S43: According to a pre-constructed data allocation mapping table, the data point set in the same sub-area is allocated as a whole to the corresponding data processing node to ensure that the data points in the same sub-area are processed on the same node, wherein the data allocation mapping table records the mapping relationship between each sub-area and the corresponding data processing node.

[0146] In this embodiment, after determining the data point set of each sub-region, it is necessary to allocate the data point set as a whole to the corresponding data processing node. This step ensures that the data points in the same sub-region are processed on the same node, avoiding the data transmission overhead across nodes and improving the parallelism and efficiency of data processing.

[0147] In the present invention, the specific implementation process of step S43 is as follows:

[0148] Data allocation mapping table: A data allocation mapping table is pre-built to record the mapping relationship between each sub-area and the corresponding data processing node.

[0149] Data point allocation: According to the data allocation mapping table, the data point set of each sub-area is allocated to the corresponding data processing node. This step ensures that the data point can be processed on the most suitable node, improving the parallelism and efficiency of data processing.

[0150] Node notification: Notify each data processing node to receive the assigned data point set and be ready to perform the corresponding data processing task. This can be achieved through message queues or RPC (remote procedure calls) and other methods.

[0151] S5: On each data processing node, the data processing tasks are executed in parallel through the distributed computing framework Spark, and then the initial results are aggregated to form the final results.

[0152] Specifically, step S5 includes the following steps:

[0153] S51: On each data processing node to which data points are allocated, an execution environment of the distributed computing framework Spark is started.

[0154] In this embodiment, efficient use of data processing nodes and parallel execution of tasks are the key to improving data processing speed and performance. To achieve this goal, the present invention proposes a method based on the distributed computing framework Spark, which ensures that each node can independently and parallelly execute data processing tasks by starting the Spark execution environment on each data processing node.

[0155] In the present invention, the specific implementation process of step S51 is as follows:

[0156] Node initialization: Start the Spark execution environment on each data processing node that is assigned a data point. This includes initializing the Spark context and setting necessary configuration parameters such as memory allocation and number of cores.

[0157] Resource check: Check the resource status on the node to ensure that there are sufficient computing resources (CPU, memory, etc.) to run Spark jobs.

[0158] Environment preparation: Load the required libraries and dependencies to ensure that the Spark execution environment can run normally.

[0159] S52: According to the requirements of the data processing task, write and deploy corresponding Spark jobs, wherein each Spark job is designed to be able to process a set of data points assigned to the node.

[0160] In this embodiment, after starting the Spark execution environment, it is necessary to write and deploy the corresponding Spark job according to the requirements of the data processing task. Each Spark job is designed to process the set of data points assigned to the node, ensuring efficient execution of the task and accuracy of the result.

[0161] In the present invention, the specific implementation process of step S52 is as follows:

[0162] Task definition: Define the specific logic of the Spark job according to the requirements of the data processing task. This includes various steps of data processing, such as data cleaning, feature extraction, and model training.

[0163] Code writing: Write the code for Spark jobs and use the APIs provided by Spark (such as RDD, DataFrame, and Dataset) to implement task logic. The code needs to specify the input data source, processing logic, and output target.

[0164] Job submission: Package the Spark job code into an executable file (such as a JAR file) and submit the job to the Spark cluster through Spark submission tools (such as spark-submit).

[0165] Resource allocation: When submitting a job, specify the required computing resources, such as the number of executors, memory and cores per executor, etc.

[0166] S53: Execute multiple Spark jobs in parallel, and collect initial output results after each Spark job is completed.

[0167] In this embodiment, after submitting a Spark job, the Spark cluster will execute multiple jobs in parallel and collect the initial output results after each job is completed. This step ensures that data processing tasks can be executed in parallel on multiple nodes, improving processing speed and efficiency.

[0168] In the present invention, the specific implementation process of step S53 is as follows:

[0169] Task Scheduling: Spark's scheduler dynamically allocates tasks based on the node load and resource availability. Each task will be assigned to one or more executors for parallel execution.

[0170] Task execution: Each executor executes the data processing logic according to the assigned task, processing the set of data points assigned to the node. During the task execution process, Spark automatically manages the status and progress of the task.

[0171] Result collection: When each task is completed, Spark collects the output results of the task. These results are usually intermediate results that need to be further aggregated and integrated.

[0172] Error handling: During task execution, if an error or exception occurs, Spark will automatically retry the task or report the error to ensure the reliability and integrity of the task.

[0173] S54: The initial results from different data processing nodes are merged and integrated through Spark's aggregation mechanism to form a globally consistent final result.

[0174] In this embodiment, after collecting the initial results of each task, it is necessary to merge and integrate these results through Spark's aggregation mechanism to form a globally consistent final result. This step ensures the integrity and accuracy of data processing and provides reliable data support for subsequent data analysis and application.

[0175] In the present invention, the specific implementation process of step S54 is as follows:

[0176] Result aggregation: Use the aggregation functions provided by Spark (such as reduceByKey, groupByKey, etc.) to merge the initial results from different data processing nodes. Aggregation functions will classify and summarize the results based on key values.

[0177] Result integration: further integrate the aggregated results to generate the final data processing results. This may include calculating statistical indicators, generating reports, and saving results to a database.

[0178] Result Verification: Verify the final results to ensure their accuracy and completeness. This can be achieved by comparing with expected results, performing verification tests, etc.

[0179] Result output: Output the final results to the specified target, such as the file system, database, visualization interface, etc., for subsequent application and analysis.

[0180] Embodiment 2

[0181] The present invention also provides a distributed data processing system for executing a distributed data processing method, referring to Figure 2 As shown, the processing system includes:

[0182] The data preprocessing module 100 is used to collect an initial data set to be processed from a target area, and use a distributed computing framework Spark to perform distributed preprocessing on the initial data set to be processed to obtain a first data set, wherein the first data set includes a plurality of data points.

[0183] The data allocation module 200 is used to dynamically and recursively divide the target area through the spatial partition tree SPT, and allocate the data points in the first data set to each divided sub-area, and add a spatial partition identifier to obtain a second data set with a spatial partition identifier.

[0184] The real-time data fusion module 300 is used to receive the real-time data stream and dynamically merge the real-time data stream into the second data set according to the spatial coordinates to obtain a third data set.

[0185] The node allocation module 400 is used to locate the position of the data point in the space partition tree SPT based on the space partition identifier, and allocate the data points in the third data set in the same sub-area to the same data processing node according to the located position.

[0186] The result aggregation module 500 is used to execute data processing tasks in parallel on each data processing node through the distributed computing framework Spark, and then aggregate the obtained initial results to form the final results.

[0187] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0188] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically-erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0189] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A distributed data processing method, characterized in that: The processing method comprises the following steps: Collecting an initial data set to be processed from a target area, and using a distributed computing framework Spark to perform distributed preprocessing on the initial data set to be processed to obtain a first data set, wherein the first data set includes a plurality of data points; The specific process of obtaining the first data set includes: Collecting an initial data set to be processed from the target area by a data acquisition component, wherein the initial data set to be processed includes multiple types of data related to the target area; In the environment of the distributed computing framework Spark, the initial data set to be processed is divided into multiple data blocks, and a preprocessing operation is performed on multiple computing nodes of Spark to obtain a preprocessed first data set; Dynamically recursively partitioning the target area through a spatial partition tree SPT, and allocating data points in the first data set to each of the divided sub-areas, while adding a spatial partition identifier to obtain a second data set with the spatial partition identifier; The specific process of obtaining the second data set includes: Constructing a spatial partition tree SPT, and subdividing the target area into finer-grained sub-areas through the constructed spatial partition tree SPT according to a preset division rule until a division stop condition is met; Traversing each data point in the first data set, and assigning each data point one by one to a sub-region located in the spatial partition tree SPT according to the spatial coordinates of each data point; During allocation, a spatial partition identifier is added to each data point allocated to the sub-region, wherein the spatial partition identifier uniquely and clearly indicates the sub-region to which the data point belongs, and is associated with a leaf node in the spatial partition tree SPT to obtain a second data set with the spatial partition identifier; receiving a real-time data stream, and dynamically integrating the real-time data stream into the second data set according to spatial coordinates to obtain a third data set; The specific process of obtaining the third data set includes: capturing a real-time data stream associated with a target area in real time, wherein the real-time data stream contains dynamically changing spatial coordinates; Preprocessing the captured real-time data stream to ensure that the real-time data stream is consistent with the data format in the second data set and the spatial coordinates are accurate; According to the spatial coordinates of each data point in the preprocessed real-time data stream, the sub-region to which it belongs is located in the spatial partition tree SPT; Dynamically inserting each data point of the located real-time data stream into the data point set of the corresponding sub-area in the second data set, and updating the data record of the associated leaf node in the spatial partition tree SPT to obtain a third data set that merges the real-time data stream; Positioning the data points in the spatial partition tree SPT based on the spatial partition identifier, and allocating the data points in the third data set in the same sub-area to the same data processing node according to the positioned positions; The allocation process specifically includes: Traverse each data point in the third data set and read the spatial partition identifier attached thereto; Using the read spatial partition identifier, locate the sub-region to which the data point belongs in the spatial partition tree SPT, and determine the set of data points in the same sub-region; According to the pre-built data allocation mapping table, the data points in the same sub-region are allocated as a whole to the corresponding data processing nodes to ensure that the data points in the same sub-region are processed on the same node, wherein the data allocation mapping table records the mapping relationship between each sub-region and the corresponding data processing node; On each data processing node, the distributed computing framework Spark is used to execute data processing tasks in parallel, and then the initial results are aggregated to form the final results. The process of forming the final result specifically includes: On each data processing node that is assigned a data point, start the execution environment of the distributed computing framework Spark; Write and deploy corresponding Spark jobs according to the requirements of data processing tasks, where each Spark job is designed to process the set of data points assigned to the node; Execute multiple Spark jobs in parallel and collect the initial results of the output after each Spark job is completed; The initial results from different data processing nodes are merged and integrated through Spark's aggregation mechanism to form a globally consistent final result.

2. A distributed data processing system for executing a distributed data processing method as claimed in claim 1, characterized in that: The processing system comprises: A data preprocessing module is used to collect an initial data set to be processed from a target area, and to perform distributed preprocessing on the initial data set to be processed using a distributed computing framework Spark to obtain a first data set, wherein the first data set includes a plurality of data points; A data allocation module, used for dynamically and recursively partitioning the target area through a spatial partition tree SPT, and allocating data points in the first data set to each of the divided sub-areas, while adding a spatial partition identifier to obtain a second data set with the spatial partition identifier; A real-time data fusion module, used for receiving a real-time data stream, and dynamically merging the real-time data stream into the second data set according to spatial coordinates to obtain a third data set; a node allocation module, configured to locate the position of the data point in the space partition tree SPT based on the space partition identifier, and allocate the data points in the third data set in the same sub-area to the same data processing node according to the located position; The result aggregation module is used to execute data processing tasks in parallel on each data processing node through the distributed computing framework Spark, and then aggregate the initial results to form the final results.

Images