A space vector polygon gap detection method based on a Spark framework

Abstract

The application discloses a kind of space vector polygon gap detection methods based on Spark framework, comprising: reading polygon space vector data and creating first RDD dataset;First RDD dataset is filtered to obtain second RDD dataset;Aggregation class and the element object in second RDD dataset are used as parameter to carry out aggregation operation, and third RDD dataset is obtained;Third RDD dataset is converted and flattened to obtain fourth RDD dataset;According to fourth RDD dataset, fifth RDD dataset is created;The data of each row of fifth RDD dataset is written to target database.The vector polygon gap detection of the application based on Spark distributed framework can be distributed in multi-machine cluster Run, also can use single machine multi-CPU parallel computing, realize efficient distributed polygon gap detection.The application also discloses a system, electronic equipment and storage medium for implementing the above method.

Description

Technical Field

[0001] This invention relates to the field of distributed data processing technology for spatial vector data, and in particular to a spatial vector polygon gap detection method based on the Spark framework. Background Technology

[0002] Polygon gap detection, as a fundamental rule in spatial vector data quality inspection, has wide applications in the Geographic Information System (GIS) field. This technology is primarily used to identify and locate uncovered spatial areas, or "gaps," between adjacent or overlapping polygons. These gaps may be caused by data acquisition errors, topological relationship errors, or logical problems in the processing. Existing vector polygon gap detection is based on single-machine serial analysis. With the expansion of the scope and improvement of the accuracy of geographic information data acquisition, the volume of vector data is growing exponentially. Taking city-level land parcel data as an example, the land parcel data of a medium-sized city typically contains hundreds of thousands to millions of polygon features. For such large-scale, high-volume spatial vector polygon data, the computational efficiency of traditional single-machine serial detection technology is low, thus failing to meet the real-time or near-real-time requirements of applications such as smart city construction, dynamic monitoring of natural resources, and emergency command. Summary of the Invention

[0003] To address the aforementioned problems in the existing technology, this invention provides a spatial vector polygon gap detection method based on the Spark framework. The technical problem to be solved by this invention is achieved through the following technical solution:

[0004] The first aspect of this invention provides a spatial vector polygon gap detection method based on the Spark framework, comprising the following steps:

[0005] Read polygonal spatial vector data and create a first RDD dataset based on the spatial vector data; wherein the first RDD dataset includes multiple partitions, each partition includes multiple rows of data, each row of data contains a feature object, and the geometric objects in the first RDD dataset are polygonal geometric objects;

[0006] The first RDD dataset is filtered to obtain the second RDD dataset by filtering out feature objects with empty geometric objects.

[0007] The second RDD dataset is aggregated using the aggregation class and the feature objects in the second RDD dataset as parameters to obtain a third RDD dataset; wherein, the third RDD dataset contains a final aggregation object;

[0008] Obtain the polygon geometry object array of the final aggregated object in the third RDD dataset to generate the target line feature object, thus obtaining the fourth RDD dataset; wherein, the fourth RDD dataset includes multiple rows of data, each row containing one target line feature object;

[0009] Each row of data in the fourth RDD dataset is used as a row of data in the fifth RDD dataset, and the spatial reference and spatial extent of the first RDD dataset are defined as the spatial reference and spatial extent of the fifth RDD dataset. The geometry type of the fifth RDD dataset is defined as line type, thus obtaining the fifth RDD dataset.

[0010] Write each row of data from the fifth RDD dataset to the target database.

[0011] In one embodiment of the present invention, the feature objects in the first RDD dataset include geometric objects and attribute objects;

[0012] The metadata information of the first RDD dataset includes: geometry type, spatial reference, and spatial extent.

[0013] In one embodiment of the present invention, the step of filtering feature objects in the first RDD dataset where the geometric objects are empty to obtain a second RDD dataset includes:

[0014] The first RDD dataset is obtained by filtering out feature objects with empty geometric objects using the filter operator in Spark.

[0015] In one embodiment of the present invention, the aggregation class includes: an aggregation function, a data fusion function, an aggregation partitioning function, and a function for obtaining aggregation results;

[0016] The step of performing an aggregation operation on the second RDD dataset using the aggregation class and feature objects from the second RDD dataset as parameters to obtain a third RDD dataset includes:

[0017] The aggregation function creates an initial aggregation object for each partition based on the spatial reference and any feature object in each partition of the second RDD dataset;

[0018] The data fusion function performs boundary fusion on the feature objects (excluding the feature objects in the initial aggregation object) and the corresponding initial aggregation object in each partition of the second RDD dataset to obtain the partition aggregation object of each partition.

[0019] The aggregation partitioning function merges the boundaries of the aggregation objects of different partitions to obtain a final aggregation object, which serves as the third RDD dataset.

[0020] The function to obtain the aggregation result is used to obtain the aggregation result of the final aggregation object and returns an array of polygon geometry objects.

[0021] In one embodiment of the present invention, obtaining an array of polygonal geometric objects of the final aggregated object in the third RDD dataset to generate a target line feature object and obtain a fourth RDD dataset includes:

[0022] The flatMap operator in Spark is used to obtain an array of polygon geometry objects of the final aggregated object in the third RDD dataset, in order to generate the target line feature object and obtain the fourth RDD dataset.

[0023] In one embodiment of the present invention, the step of obtaining an array of polygonal geometry objects of the final aggregated object in the third RDD dataset through the flatMap operator in Spark to generate target line feature objects and obtain a fourth RDD dataset includes:

[0024] The array of polygon geometry objects of the final aggregated object in the third RDD dataset is obtained by using the flatMap operator in Spark, and the geometric boundary coordinate data of the array of polygon geometry objects is extracted to generate the target line geometry object. Then, the target line feature object is generated based on the target line geometry object. The target line feature object includes the target line geometry object and an empty attribute object.

[0025] In one embodiment of the present invention, each row of data in the fifth RDD dataset contains a line feature, the line feature including the target line geometry object and an empty attribute object.

[0026] A second aspect of this invention provides a spatial vector polygon gap detection system based on the Spark framework, comprising:

[0027] The data reading module is used to read polygonal spatial vector data and create a first RDD dataset based on the spatial vector data; wherein, the first RDD dataset includes multiple partitions, each partition includes multiple rows of data, each row of data contains a feature object, and the geometric objects in the first RDD dataset are polygonal geometric objects;

[0028] The data filtering module is used to filter feature objects in the first RDD dataset that have empty geometric objects to obtain the second RDD dataset.

[0029] The data aggregation module is used to perform aggregation operations on the second RDD dataset using the aggregation class and the feature objects in the second RDD dataset as parameters to obtain a third RDD dataset; wherein, the third RDD dataset contains a final aggregation object;

[0030] The boundary extraction module is used to obtain an array of polygonal geometric objects of the final aggregated object in the third RDD dataset to generate a target line feature object, thus obtaining a fourth RDD dataset; wherein, the fourth RDD dataset includes multiple rows of data, and each row of data contains a target line feature object;

[0031] The data reconstruction module is used to take each row of data in the fourth RDD dataset as each row of data in the fifth RDD dataset, define the spatial reference and spatial range of the first RDD dataset as the spatial reference and spatial range of the fifth RDD dataset, define the geometric type of the fifth RDD dataset as line type, and obtain the fifth RDD dataset.

[0032] The data writing module is used to write each row of data from the fifth RDD dataset to the target database.

[0033] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a spatial vector polygon gap detection method based on the Spark framework provided in the first aspect of the present invention.

[0034] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a spatial vector polygon gap detection method based on the Spark framework provided in the first aspect of the present invention.

[0035] The beneficial effects of this invention are:

[0036] This invention enables distributed operation of vector polygon gap detection based on the Spark distributed framework in multi-machine clusters, or in parallel computation using multiple CPUs on a single machine. It effectively utilizes computing resources and can analyze and process large-scale, high-volume vector spatial data, while also offering high efficiency for small-scale, low-volume data. By combining Spark distributed aggregation operators and customizing the aggregation process, data transformation overhead is effectively reduced, achieving efficient distributed polygon gap detection.

[0037] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0038] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0039] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0040] Figure 1 A schematic flowchart of a spatial vector polygon gap detection method based on the Spark framework provided in an embodiment of the present invention;

[0041] Figure 2 This is a block diagram of a spatial vector polygon gap detection system based on the Spark framework, provided as an embodiment of the present invention. Detailed Implementation

[0042] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0043] like Figure 1 As shown, the first aspect of this invention provides a spatial vector polygon gap detection method based on the Spark framework, comprising the following steps:

[0044] Step 11: Read the polygon spatial vector data and create the first RDD dataset based on the spatial vector data.

[0045] The first RDD dataset includes multiple partitions, each partition includes multiple rows of data, and each row of data contains a feature object. The geometric objects in the first RDD dataset are polygonal geometric objects.

[0046] Step 12: Filter the feature objects in the first RDD dataset that have empty geometric objects to obtain the second RDD dataset.

[0047] Step 13: Use the aggregation class and the feature objects in the second RDD dataset as parameters to perform an aggregation operation on the second RDD dataset to obtain the third RDD dataset.

[0048] The third RDD dataset contains a final aggregate object.

[0049] Step 14: Obtain the polygon geometry object array of the final aggregated object in the third RDD dataset to generate the target line feature object, and perform flattening processing to obtain the fourth RDD dataset.

[0050] The fourth RDD dataset consists of multiple rows of data, each containing a target line feature object.

[0051] Step 15: Take each row of data from the fourth RDD dataset as each row of data from the fifth RDD dataset, define the spatial reference and spatial extent of the first RDD dataset as the spatial reference and spatial extent of the fifth RDD dataset, define the geometry type of the fifth RDD dataset as line type, and obtain the fifth RDD dataset.

[0052] Step 16: Write each row of data from the fifth RDD dataset to the target database.

[0053] In this embodiment, polygonal spatial vector data is read based on the Spark distributed computing framework (Apache Spark is a fast and general-purpose computing engine designed for large-scale data processing) and HDFS (Hadoop Distributed File System). It supports distributed reading and writing of Shapefile spatial data in local storage or HDFS, and can also read and write spatial data in the PostGIS database (PostGIS is a spatial extension plugin for the PostgreSQL relational database that integrates Geographic Information System (GIS) functionality into the database, enabling users to efficiently store, query, analyze, and manipulate spatial data) based on database connection information. Vector polygon gap detection based on the Spark distributed framework can run in a distributed manner in a multi-machine cluster, or use single-machine multi-CPU parallel computing, effectively utilizing computing resources. It can analyze and process large-scale, large-volume vector spatial data, and also has advantages in high analysis efficiency for small-scale, small-volume data. Combined with Spark distributed aggregation operators, a customized aggregation process is used, effectively reducing data transformation overhead and achieving efficient distributed polygon gap detection.

[0054] Compared to traditional single-machine serial vector polygon gap detection technology, this invention can run in a distributed manner in a multi-machine cluster, or use a single machine with multiple CPUs for parallel computing, effectively utilizing computing resources. Traditional single-machine serial vector polygon gap detection can only analyze and process small-scale, small-volume spatial vector data, while this invention can analyze and process large-scale, large-volume vector spatial data, and also has the advantage of high analysis efficiency for small-scale, small-volume data. Traditional vector polygon gap detection platforms can only read data from the local file system, while this invention, based on the Spark and Hadoop distributed read / write system, supports reading and writing vector data in NFS and HDFS.

[0055] Based on the first aspect of the present invention, the second aspect of the present invention provides a more detailed description of a spatial vector polygon gap detection method based on the Spark framework. The second aspect of the present invention provides a spatial vector polygon gap detection method based on the Spark framework, comprising the following steps:

[0056] Step 21: Read the polygon spatial vector data and create the first RDD dataset (Resilient Distributed Dataset) based on the spatial vector data.

[0057] The system reads polygon spatial vector data based on Spark distributed computing and the Hadoop distributed file read / write system. It supports reading polygon vector data from local or HDFS Shapefiles, and can also read polygon spatial vector data from PostGIS based on database connection information. The read data is stored in multiple partitions, and the number of partitions can be customized according to machine resources. When machine CPU and memory resources are sufficient, the larger the data volume and the more partitions, the faster the calculation speed.

[0058] The first RDD dataset is a custom FeatureRDD, which represents an RDD that stores feature data. The first RDD dataset includes multiple partitions, each containing multiple rows of data. Each row contains a feature object, which includes polygon geometry objects and attribute objects. The metadata information of the first RDD dataset includes: geometry type, spatial reference, and spatial extent.

[0059] Step 22: Filter the feature objects in the first RDD dataset that have empty geometric objects to obtain the second RDD dataset.

[0060] In this step, the `filter` operator in Spark is used to filter out features with empty geometric objects from the first RDD dataset, forming the second RDD dataset. All features in the second RDD dataset have non-empty geometric objects, and its structure is identical to the first RDD dataset.

[0061] Step 23: Use the aggregation class and the feature objects in the second RDD dataset as parameters to perform an aggregation operation on the second RDD dataset to obtain the third RDD dataset.

[0062] The third RDD dataset contains a final aggregate object.

[0063] The aggregation class StitchCombiner is used to aggregate features. The aggregation class mainly includes the aggregation function combiner, the data merging function mergeValue, the aggregation partitioning function mergeCombiner, and the function getResult to get the aggregation result.

[0064] Step 23 includes steps 213-234:

[0065] Step 231: The aggregation function creates an initial aggregation object for each partition based on any feature object in each partition of the spatial reference and the second RDD dataset.

[0066] In this step, the aggregation function is used to create a new aggregation object instance. For each partition, the aggregation function accepts a spatial reference object and a feature object from the partition as parameters, and returns a new aggregation object. Creating a new aggregation object instance for each partition is equivalent to creating an initial aggregation object.

[0067] Step 232: The data fusion function merges the feature objects (excluding the feature objects in the initial aggregation object) in each partition of the second RDD dataset with the corresponding initial aggregation object to obtain the partition aggregation object for each partition.

[0068] The merge data function accepts an aggregation object and a feature object. It executes within a single partition, merging the polygon geometry of the features in the aggregation object and the input feature object by boundary matching. Adjacent polygons are merged into a single polygon, while non-adjacent polygons are not merged. Finally, it returns a new aggregation object. Adjacent polygons refer to polygons whose boundaries coincide and whose regions overlap.

[0069] In this step, for a single partition, the boundary fusion is performed on the initial aggregation object and the feature objects other than the feature objects in step 231. Specifically, the aggregation logic of the data fusion function is to accept the initial aggregation object and a feature object as parameters, perform boundary fusion on the polygon geometry of the two, and return an intermediate aggregation object (intermediate result). Then, the intermediate result and the next feature object are used as parameters for boundary fusion, and the returned result is then performed with the next feature object, and so on, until all feature objects in the current partition have completed the boundary fusion operation, and a partition aggregation object for the current partition is returned. Boundary fusion is performed on each partition separately, and each partition obtains a partition aggregation object.

[0070] Here, the boundary fusion operation is implemented using an existing geometric fusion algorithm. For example, during fusion, a spatial index is first constructed to find the geometric data that intersects with the minimum geometric boundaries. Then, a topological edge graph is constructed for all the search results, the usage frequency of the edges is counted, internal edges are removed based on the usage frequency, and a geometric contour is constructed.

[0071] Step 233: The aggregation partitioning function merges the partition aggregation objects of different partitions to obtain a final aggregation object, which serves as the third RDD dataset.

[0072] In this step, the aggregation partitioning function accepts two aggregation objects as parameters. When the function is executed, it merges the geometry of the aggregation result elements in the two aggregation objects. If the boundaries are adjacent, the merged boundary is a polygon. If the boundaries are not adjacent, they are not merged. Finally, it returns a new aggregation object.

[0073] The aggregation logic of the aggregation partitioning function is as follows: it accepts the aggregation objects of the first and second partitions as parameters, performs boundary fusion on the geometric features of the aggregation results in the two aggregation objects, and returns a single result (also an aggregation object). That is, it sequentially takes the aggregation objects of the two partitions as parameters, performs boundary fusion, returns a result, then takes the returned result as a parameter to return a secondary result, and so on, until the final result consists of only one aggregation object. For example, the aggregation logic is shown below:

[0074] Aggregate partition function (partition1 aggregate object, partition2 aggregate object) → result A aggregate object;

[0075] Aggregate partition function(partition3 aggregate object, partition4 aggregate object) → result B aggregate object;

[0076] Aggregate partition function(partition5 aggregate object, partition6 aggregate object) → result C aggregate object;

[0077] Aggregate partition function (partition 7 aggregate object, partition 8 aggregate object) → result D aggregate object;

[0078] Aggregate partitioning function (Result A aggregate object, Result B aggregate object) → Result E aggregate object;

[0079] Aggregate partitioning function (result C aggregate object, result D aggregate object) → result F aggregate object;

[0080] Aggregate partitioning function (result E aggregate object, result F aggregate object) → final result aggregate object.

[0081] or

[0082] Aggregate partition function (partition1 aggregate object, partition2 aggregate object) → result A aggregate object;

[0083] Aggregate partition function(partition3 aggregate object, partition4 aggregate object) → result B aggregate object;

[0084] Aggregate partition function(partition5 aggregate object, partition6 aggregate object) → result C aggregate object;

[0085] Aggregate partitioning function (Result A aggregate object, Result B aggregate object) → Result D aggregate object;

[0086] Aggregate partitioning function (result C aggregate object, result D aggregate object) → final result aggregate object.

[0087] Here, the boundary fusion operation is implemented using an existing geometric fusion algorithm.

[0088] Step 234: The function to obtain the aggregation result is used to obtain the aggregation result of the aggregation object, returning an array of polygon geometry objects.

[0089] Feature aggregation is based on the combineByKey operator in Spark. It processes the second RDD dataset, takes a custom aggregation class and feature objects as parameters, uses the Spark distributed framework for computation, and finally returns a third RDD dataset, which contains an aggregation object.

[0090] Step 24: Obtain the polygon geometry object array of the final aggregated object in the third RDD dataset to generate the target line feature object and obtain the fourth RDD dataset.

[0091] The fourth RDD dataset consists of multiple rows of data, each containing a target line feature object.

[0092] Specifically, the `flatMap` operator in Spark is used to obtain an array of polygon geometry objects from the final aggregated object in the third RDD dataset. The geometric boundary coordinates of each polygon geometry object are extracted to generate target line geometry objects. Then, target line feature objects are generated based on these target line geometry objects. Next, a flattening process is performed, converting all target line feature objects into multiple rows of data, each row containing one target line feature object. Each target line feature object includes a target line geometry object and an empty attribute object.

[0093] Step 25: Define the spatial reference and spatial extent of the first RDD dataset as the spatial reference and spatial extent of the fifth RDD dataset, define the geometry type of the fifth RDD dataset as line type, and use each row of data from the fourth RDD dataset as data for the fifth RDD dataset.

[0094] This step involves data reconstruction, converting the fourth RDD into a custom Feature RDD to facilitate data writing and adaptation. Each row of data in the fifth RDD dataset contains a line feature, which includes the target line geometry object and an empty attribute object. The metadata information of the fifth RDD includes information such as the geometric type, spatial reference, and spatial extent of the spatial data.

[0095] Step 26: Write each row of data from the fifth RDD dataset to the target database.

[0096] Using the foreachPartition function in Spark and related Hadoop files, a class is written to write the fifth RDD dataset to a local or HDFS Shapefile. Alternatively, the fifth RDD dataset can be written to a PostGIS spatial database based on the database connection information. The written data is the gap data for spatial polygon data detection.

[0097] In this implementation, the problem of gap detection in large-scale spatial vector polygon data cannot be completed in a short time is addressed by parallel processing, which solves the problems of low efficiency and long processing time of traditional serial analysis. Utilizing the Spark distributed computing framework, parallel analysis can be performed on a single machine or multi-machine cluster, making full use of computing resources, accelerating processing speed, and making it easier to handle large-scale spatial vector data.

[0098] In one feasible implementation, the above method is based on the Spark distributed framework and can be developed using the Scala language. In addition, PySpark is a Spark interface library developed in Python, providing a set of functions for large-scale data processing. PySpark can utilize Spark parallel technology to read vector data from local storage or HDFS, as well as vector data from PostGIS databases, and perform vector polygon gap detection. Furthermore, the computational logic of the above method can be written in Python, improving its versatility and scalability.

[0099] like Figure 2 As shown, a third aspect of the present invention provides a spatial vector polygon gap detection system based on the Spark framework, comprising:

[0100] The data reading module 31 is used to read polygonal spatial vector data and create a first RDD dataset based on the spatial vector data; wherein, the first RDD dataset includes multiple partitions, each partition includes multiple rows of data, each row of data contains a feature object, and the geometric objects in the first RDD dataset are polygonal geometric objects;

[0101] Data filtering module 32 is used to filter feature objects in the first RDD dataset that have empty geometric objects to obtain the second RDD dataset;

[0102] The data aggregation module 33 is used to perform aggregation operations on the second RDD dataset by taking the aggregation class and the feature objects in the second RDD dataset as parameters to obtain the third RDD dataset; wherein the third RDD dataset contains a final aggregation object;

[0103] Boundary extraction module 34 is used to obtain an array of polygon geometry objects of the final aggregated object in the third RDD dataset to generate target line feature objects and obtain the fourth RDD dataset; wherein, the fourth RDD dataset includes multiple rows of data, each row of data contains one target line feature object;

[0104] The data reconstruction module 35 is used to take each row of data in the fourth RDD dataset as each row of data in the fifth RDD dataset, define the spatial reference and spatial extent of the first RDD dataset as the spatial reference and spatial extent of the fifth RDD dataset, define the geometry type of the fifth RDD dataset as line type, and obtain the fifth RDD dataset.

[0105] The data writing module 36 is used to write each row of data from the fifth RDD dataset to the target database.

[0106] In one embodiment of the present invention, the feature objects in the first RDD dataset include geometric objects and attribute objects;

[0107] The metadata information for the first RDD dataset includes: geometry type, spatial reference, and spatial extent.

[0108] In one embodiment of the present invention, feature objects in the first RDD dataset that have empty geometric objects are filtered to obtain a second RDD dataset, including:

[0109] The second RDD dataset is obtained by filtering out feature objects with empty geometric objects in the first RDD dataset using the filter operator in Spark.

[0110] In one embodiment of the present invention, the aggregation class includes: an aggregation function, a data fusion function, an aggregation partitioning function, and a function for obtaining aggregation results;

[0111] The aggregation operation on the second RDD dataset is performed using the aggregation class and feature objects from the second RDD dataset as parameters to obtain the third RDD dataset, which includes:

[0112] The aggregation function creates an initial aggregation object for each partition based on any feature object in each partition of the spatial reference and the second RDD dataset;

[0113] The data fusion function merges the feature objects (excluding those in the initial aggregation object) in each partition of the second RDD dataset with the corresponding initial aggregation object to obtain the partition aggregation object for each partition.

[0114] The aggregation partitioning function merges the partition aggregation objects from different partitions to obtain a final aggregation object, which serves as the third RDD dataset.

[0115] The function to retrieve the aggregation result is used to obtain the aggregation result of the final aggregation object, and returns an array of polygon geometry objects.

[0116] In one embodiment of the present invention, an array of polygonal geometric objects of the final aggregated object in the third RDD dataset is obtained to generate target line feature objects, resulting in a fourth RDD dataset, including:

[0117] The flatMap operator in Spark is used to obtain an array of polygon geometry objects from the final aggregated objects in the third RDD dataset, which is then used to generate target line feature objects, resulting in the fourth RDD dataset.

[0118] In one embodiment of the present invention, the polygon geometry object array of the final aggregated object in the third RDD dataset is obtained through the flatMap operator in Spark to generate the target line feature object, thus obtaining the fourth RDD dataset, including:

[0119] The flatMap operator in Spark is used to obtain the array of polygon geometry objects of the final aggregated object in the third RDD dataset, and the geometric boundary coordinate data of the polygon geometry object array is extracted to generate the target line geometry object. Then, the target line feature object is generated based on the target line geometry object. The target line feature object includes the target line geometry object and the empty attribute object.

[0120] In one embodiment of the present invention, each row of data in the fifth RDD dataset contains a line feature, which includes a target line geometry object and an empty attribute object.

[0121] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the spatial vector polygon gap detection method based on the Spark framework provided in the above-described embodiments of the present invention.

[0122] The fifth aspect of this invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the spatial vector polygon gap detection method based on the Spark framework provided in the above-described embodiments of this invention.

[0123] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage system located remotely from the aforementioned processor.

[0124] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware systems.

[0125] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0126] For system / electronic device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be found in the description of the method embodiments.

[0127] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0128] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0129] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0130] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A spatial vector polygon gap detection method based on the Spark framework, characterized in that, Includes the following steps: Read polygonal spatial vector data and create a first RDD dataset based on the spatial vector data; wherein the first RDD dataset includes multiple partitions, each partition includes multiple rows of data, each row of data contains a feature object, and the geometric objects in the first RDD dataset are polygonal geometric objects; The first RDD dataset is filtered to obtain the second RDD dataset by filtering out feature objects with empty geometric objects. The second RDD dataset is aggregated using the aggregation class and the feature objects in the second RDD dataset as parameters to obtain a third RDD dataset; wherein, the third RDD dataset contains a final aggregation object; Obtain the polygon geometry object array of the final aggregated object in the third RDD dataset to generate the target line feature object, thus obtaining the fourth RDD dataset; wherein, the fourth RDD dataset includes multiple rows of data, each row containing one target line feature object; Each row of data in the fourth RDD dataset is used as a row of data in the fifth RDD dataset, and the spatial reference and spatial extent of the first RDD dataset are defined as the spatial reference and spatial extent of the fifth RDD dataset. The geometry type of the fifth RDD dataset is defined as line type, thus obtaining the fifth RDD dataset. Write each row of data from the fifth RDD dataset to the target database; The feature objects in the first RDD dataset include geometric objects and attribute objects; The metadata information of the first RDD dataset includes: geometry type, spatial reference, and spatial extent; The aggregation class includes: aggregation functions, data fusion functions, aggregation partitioning functions, and functions for obtaining aggregation results; The step of performing an aggregation operation on the second RDD dataset using the aggregation class and feature objects from the second RDD dataset as parameters to obtain a third RDD dataset includes: The aggregation function creates an initial aggregation object for each partition based on the spatial reference and any feature object in each partition of the second RDD dataset; The data fusion function performs boundary fusion on the feature objects (excluding the feature objects in the initial aggregation object) and the corresponding initial aggregation object in each partition of the second RDD dataset to obtain the partition aggregation object of each partition. The aggregation partitioning function merges the boundaries of the aggregation objects of different partitions to obtain a final aggregation object, which serves as the third RDD dataset. The function to obtain the aggregation result is used to obtain the aggregation result of the final aggregation object and returns an array of polygon geometry objects.

2. The method as described in claim 1, characterized in that, The step of filtering feature objects with empty geometric objects in the first RDD dataset to obtain the second RDD dataset includes: The first RDD dataset is obtained by filtering out feature objects with empty geometric objects using the filter operator in Spark.

3. The method as described in claim 1, characterized in that, The step of obtaining the polygon geometry object array of the final aggregated object in the third RDD dataset to generate the target line feature object, thus obtaining the fourth RDD dataset, includes: The flatMap operator in Spark is used to obtain an array of polygon geometry objects of the final aggregated object in the third RDD dataset, in order to generate the target line feature object and obtain the fourth RDD dataset.

4. The method as described in claim 3, characterized in that, The process involves obtaining an array of polygon geometry objects from the final aggregated object in the third RDD dataset using the flatMap operator in Spark, to generate target line feature objects, resulting in a fourth RDD dataset, which includes: The array of polygon geometry objects of the final aggregated object in the third RDD dataset is obtained by using the flatMap operator in Spark, and the geometric boundary coordinate data of the array of polygon geometry objects is extracted to generate the target line geometry object. Then, the target line feature object is generated based on the target line geometry object. The target line feature object includes the target line geometry object and an empty attribute object.

5. The method as described in claim 4, characterized in that, Each row of data in the fifth RDD dataset contains a line feature, which includes the target line geometry object and an empty attribute object.

6. A spatial vector polygon gap detection system based on the Spark framework, characterized in that, include: The data reading module is used to read polygonal spatial vector data and create a first RDD dataset based on the spatial vector data; wherein, the first RDD dataset includes multiple partitions, each partition includes multiple rows of data, each row of data contains a feature object, and the geometric objects in the first RDD dataset are polygonal geometric objects; The data filtering module is used to filter feature objects in the first RDD dataset that have empty geometric objects to obtain the second RDD dataset. The data aggregation module is used to perform aggregation operations on the second RDD dataset using the aggregation class and the feature objects in the second RDD dataset as parameters to obtain a third RDD dataset; wherein, the third RDD dataset contains a final aggregation object; The boundary extraction module is used to obtain an array of polygonal geometric objects of the final aggregated object in the third RDD dataset to generate a target line feature object, thus obtaining a fourth RDD dataset; wherein, the fourth RDD dataset includes multiple rows of data, and each row of data contains a target line feature object; The data reconstruction module is used to take each row of data in the fourth RDD dataset as each row of data in the fifth RDD dataset, define the spatial reference and spatial range of the first RDD dataset as the spatial reference and spatial range of the fifth RDD dataset, define the geometric type of the fifth RDD dataset as line type, and obtain the fifth RDD dataset. The data writing module is used to write each row of data from the fifth RDD dataset to the target database; The feature objects in the first RDD dataset include geometric objects and attribute objects; The metadata information of the first RDD dataset includes: geometry type, spatial reference, and spatial extent; The aggregation class includes: aggregation functions, data fusion functions, aggregation partitioning functions, and functions for obtaining aggregation results; The step of performing an aggregation operation on the second RDD dataset using the aggregation class and feature objects from the second RDD dataset as parameters to obtain a third RDD dataset includes: The aggregation function creates an initial aggregation object for each partition based on the spatial reference and any feature object in each partition of the second RDD dataset; The data fusion function performs boundary fusion on the feature objects (excluding the feature objects in the initial aggregation object) and the corresponding initial aggregation object in each partition of the second RDD dataset to obtain the partition aggregation object of each partition. The aggregation partitioning function merges the boundaries of the aggregation objects of different partitions to obtain a final aggregation object, which serves as the third RDD dataset. The function to obtain the aggregation result is used to obtain the aggregation result of the final aggregation object and returns an array of polygon geometry objects.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the spatial vector polygon gap detection method based on the Spark framework as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the spatial vector polygon gap detection method based on the Spark framework as described in any one of claims 1 to 5.

Images