Space vector polygon gap detection method based on Spark framework

Abstract

The invention discloses a space vector polygon gap detection method based on a Spark framework. The method comprises the following steps: reading polygon space vector data and creating a first RDD data set; filtering the first RDD data set to obtain a second RDD data set; performing aggregation operation by taking the aggregation class and the element objects in the second RDD data set as parameters to obtain a third RDD data set; performing conversion and flattening processing on the third RDD data set to obtain a fourth RDD data set; creating a fifth RDD data set according to the fourth RDD data set; and writing each row of data of the fifth RDD data set to the target database. According to the method, distributed operation in a multi-machine cluster can be realized through vector polygon gap detection based on a Spark distributed framework, and high-efficiency distributed polygon gap detection can be realized by using single-machine multi-CPU (Central Processing Unit) parallel calculation. The invention further discloses a system for implementing the method, electronic equipment and a storage medium.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed data processing of spatial vector data, and particularly relates to a spatial vector polygon gap detection method based on a Spark framework. BACKGROUND

[0002] Polygon gap detection serves as a basic rule in spatial vector data quality inspection and has a wide application in the field of geographic information systems (GIS). This technology is mainly used to identify and locate the uncovered spatial regions between adjacent or overlapping polygons, i.e., "gaps", which may be caused by data collection errors, topological relationship errors or logical problems in the processing process. Existing vector polygon gap detection is based on single-machine serial operation analysis. With the expansion of the range and improvement of the precision of geographic information data collection, the amount of vector data presents an exponential growth. Taking municipal land parcel data as an example, the land parcel data of a medium-sized city usually contains hundreds of thousands to millions of polygon elements. For such large-scale and large-volume spatial vector polygon data, the calculation efficiency of the traditional single-machine serial detection technology is low, and thus it cannot meet the real-time or quasi-real-time requirements in the application scenarios of smart city construction, natural resource dynamic monitoring, emergency command, etc. SUMMARY

[0003] In order to solve the above problems existing in the prior art, the present application provides a spatial vector polygon gap detection method based on a Spark framework. The technical problems to be solved by the present application are realized through the following technical solutions: The first aspect of the embodiment of the present application provides a spatial vector polygon gap detection method based on a Spark framework, comprising the following steps: reading polygon spatial vector data and creating a first RDD dataset according to the spatial vector data; wherein the first RDD dataset includes multiple partitions, each partition includes multiple rows of data, each row of data contains an element object, and the geometric object in the first RDD dataset is a polygon geometric object; filtering the element objects with empty geometric objects in the first RDD dataset to obtain a second RDD dataset; performing an aggregation operation on the second RDD dataset by taking the aggregation class and the element objects in the second RDD dataset as parameters to obtain a third RDD dataset; wherein the third RDD dataset contains a final aggregation object; obtaining the polygon geometric object array of the final aggregation object in the third RDD dataset to generate a target line element object, and obtaining a fourth RDD dataset; wherein the fourth RDD dataset includes multiple rows of data, and each row of data contains a target line element object; each row of data of the fourth RDD dataset as each row of data of a fifth RDD dataset, and defining the spatial reference and spatial extent of the first RDD dataset as the spatial reference and spatial extent of the fifth RDD dataset, defining the geometry type of the fifth RDD dataset as a line type, to obtain the fifth RDD dataset; writing each row of data of the fifth RDD dataset to a target database.

[0004] In an embodiment of the present application, the feature objects in the first RDD dataset include geometry objects and attribute objects. The metadata information of the first RDD dataset includes a geometry type, a spatial reference, and a spatial extent.

[0005] In an embodiment of the present application, the filtering processing of the feature objects with empty geometry objects in the first RDD dataset to obtain the second RDD dataset includes: filtering processing of the feature objects with empty geometry objects in the first RDD dataset by using a filter operator in Spark to obtain the second RDD dataset.

[0006] In an embodiment of the present application, the aggregation class includes an aggregation function, a fusion data function, an aggregation partition function, and an acquisition of aggregation result function. The aggregation operation on the second RDD dataset by using the aggregation class and the feature objects in the second RDD dataset as parameters to obtain the third RDD dataset includes: The aggregation function creates an initial aggregation object of each partition according to the spatial reference and any one feature object in each partition of the second RDD dataset; The fusion data function performs boundary fusion on the feature objects in each partition of the second RDD dataset except the feature objects in the initial aggregation objects and the corresponding initial aggregation objects to obtain a partition aggregation object of each partition; The aggregation partition function performs boundary fusion on the partition aggregation objects of different partitions to obtain a final aggregation object as the third RDD dataset; The acquisition of aggregation result function is used to acquire the aggregation result of the final aggregation object, and returns an array of polygon geometry objects.

[0007] In an embodiment of the present application, the acquisition of the array of polygon geometry objects of the final aggregation object in the third RDD dataset to generate a target line feature object to obtain the fourth RDD dataset includes: The polygon geometry object array of the final aggregation object in the third RDD dataset is obtained through a flatMap operator in Spark to generate a target line feature object, and a fourth RDD dataset is obtained.

[0008] In an embodiment of the present application, the polygon geometry object array of the final aggregation object in the third RDD dataset is obtained through a flatMap operator in Spark to generate a target line feature object, and a fourth RDD dataset is obtained, including: The polygon geometry object array of the final aggregation object in the third RDD dataset is obtained through a flatMap operator in Spark, and the geometric boundary coordinate data of the polygon geometry object array is extracted to generate a target line geometry object, and a target line feature object is generated according to the target line geometry object; wherein the target line feature object includes the target line geometry object and an empty attribute object.

[0009] In an embodiment of the present application, each row of data in the fifth RDD dataset contains a line feature, and the line feature includes the target line geometry object and an empty attribute object.

[0010] The second aspect of the embodiment of the present application provides a spatial vector polygon gap detection system based on a Spark framework, including: A data reading module is configured to read polygon spatial vector data and create a first RDD dataset according to the spatial vector data; wherein the first RDD dataset includes multiple partitions, each partition includes multiple rows of data, each row of data contains a feature object, and the geometric object in the first RDD dataset is a polygon geometric object; A data filtering module is configured to filter the feature object with an empty geometric object in the first RDD dataset to obtain a second RDD dataset; A data aggregation module is configured to perform an aggregation operation on the second RDD dataset by taking the aggregation class and the feature object in the second RDD dataset as parameters to obtain a third RDD dataset; wherein the third RDD dataset contains a final aggregation object; A boundary extraction module is configured to obtain the polygon geometry object array of the final aggregation object in the third RDD dataset to generate a target line feature object, and obtain a fourth RDD dataset; wherein the fourth RDD dataset includes multiple rows of data, and each row of data contains a target line feature object; a data reconstruction module, configured to reconstruct each row of data in the fourth RDD dataset as each row of data in a fifth RDD dataset, define the spatial reference and spatial extent of the first RDD dataset as the spatial reference and spatial extent of the fifth RDD dataset, and define the geometry type of the fifth RDD dataset as a line type, to obtain the fifth RDD dataset; a data writing module, configured to write each row of data in the fifth RDD dataset to a target database.

[0011] A third aspect of the embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method for detecting a spatial vector polygon gap based on a Spark framework provided in the first aspect of the embodiment of the present application when executing the program.

[0012] A fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the method for detecting a spatial vector polygon gap based on a Spark framework provided in the first aspect of the embodiment of the present application.

[0013] The present application has the following beneficial effects: The method for detecting a spatial vector polygon gap based on a Spark distributed framework can be distributedly executed in a multi-machine cluster, and can also be executed in a single-machine multi-CPU parallel computing mode, so that the computing resources are effectively utilized, and a large range and a large amount of vector spatial data can be analyzed and processed, and the method also has advantages such as high analysis efficiency for small range and small amount of data. In combination with a Spark distributed aggregation operator, an aggregation process is customized, so that data conversion consumption is effectively reduced, and efficient distributed polygon gap detection is realized.

[0014] Other features and advantages of the present application will be described in the following description, and some will become apparent from the description, or will be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the written description, claims, and drawings.

[0015] The technical solutions of the present application will be further described in detail below with the help of the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, and are used to explain the present application together with the embodiments of the present application, and do not constitute a limitation of the present application. In the drawings: Figure 1 A flowchart of the method for detecting a spatial vector polygon gap based on a Spark framework provided in the embodiment of the present application; Figure 2 A block diagram of a space vector polygon gap detection system based on a Spark framework is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0017] The present application will be further described below in conjunction with specific embodiments, but the embodiments of the present application are not limited thereto.

[0018] As shown in Figure 1 The first aspect of the embodiment of the present application provides a space vector polygon gap detection method based on a Spark framework, comprising the following steps: Step 11, reading the polygon space vector data, and creating a first RDD dataset according to the space vector data.

[0019] The first RDD dataset includes multiple partitions, each partition includes multiple rows of data, each row of data contains a feature object, and the geometric object in the first RDD dataset is a polygon geometric object.

[0020] Step 12, filtering the feature objects with empty geometric objects in the first RDD dataset to obtain a second RDD dataset.

[0021] Step 13, performing aggregation operation on the second RDD dataset by taking the aggregation class and the feature objects in the second RDD dataset as parameters to obtain a third RDD dataset.

[0022] The third RDD dataset contains a final aggregation object.

[0023] Step 14, obtaining the polygon geometric object array of the final aggregation object in the third RDD dataset to generate a target line feature object, and performing flattening processing to obtain a fourth RDD dataset.

[0024] The fourth RDD dataset includes multiple rows of data, each row of data contains a target line feature object.

[0025] Step 15, taking each row of data of the fourth RDD dataset as each row of data of a fifth RDD dataset, defining the spatial reference and spatial range of the first RDD dataset as the spatial reference and spatial range of the fifth RDD dataset, and defining the geometric type of the fifth RDD dataset as a line type to obtain the fifth RDD dataset.

[0026] Step 16, writing each row of data of the fifth RDD dataset to a target database.

[0027] In this embodiment, polygonal spatial vector data is read based on the Spark distributed computing framework (Apache Spark is a fast and versatile computing engine designed for large-scale data processing) and HDFS (Hadoop Distributed File System). This supports distributed reading and writing of Shapefile spatial data stored locally or in HDFS. It can also read and write spatial data from a PostGIS database (PostGIS is a spatial extension plugin for the PostgreSQL relational database, integrating geographic information system (GIS) functionality into the database, enabling users to efficiently store, query, analyze, and manipulate spatial data) based on database connection information. Vector polygon gap detection based on the Spark distributed framework can be distributed across a multi-machine cluster or run using multiple CPUs on a single machine, effectively utilizing computing resources. This allows for analysis and processing of large-scale, high-volume vector spatial data, while also offering high analysis efficiency for smaller-scale, low-volume data. Incorporating Spark's distributed aggregation operators and customizing the aggregation process effectively reduces data conversion overhead and enables efficient distributed polygon gap detection.

[0028] Compared with the traditional single-machine serial vector polygon gap detection technology, the present invention can run in a distributed manner in a multi-machine cluster, and can also use single-machine multi-CPU parallel computing to effectively utilize computing resources. The traditional single-machine serial vector polygon gap detection can only analyze and process small-scale, small-volume spatial vector data, while the present invention can analyze and process large-scale, large-volume vector spatial data, and also has the advantages of high analysis efficiency for small-scale, small-volume data. The traditional vector polygon gap detection platform can only read local file system data, while the present invention is based on the Spark and Hadoop distributed read and write system, and supports reading and writing vector data in NFS and HDFS.

[0029] On the basis of the first aspect of the embodiment of the present invention, the second aspect of the embodiment of the present invention further describes in detail a method for detecting gaps in a space vector polygon based on a Spark framework. The second aspect of the embodiment of the present invention provides a method for detecting gaps in a space vector polygon based on a Spark framework, comprising the following steps: Step 21: read polygonal spatial vector data and create a first RDD (Resilient Distributed Dataset) data set based on the spatial vector data.

[0030] The reading of the polygon spatial vector data is based on Spark distributed computing and Hadoop distributed file reading and writing system, supports reading Shapefile polygon vector data in local or HDFS, and can also read polygon spatial vector data in PostGIS according to database connection information. The read data exists in multiple partitions, the number of partitions can be customized according to machine resources, when the machine CPU and memory resources are sufficient, the larger the data volume, the more the number of partitions, the faster the calculation speed.

[0031] The first RDD dataset is a custom FeatureRDD, FeatureRDD represents an RDD storing feature data, the first RDD dataset includes multiple partitions, each partition includes multiple rows of data, each row of data contains a feature object, the feature object includes a polygon geometry object and an attribute object. The metadata information of the first RDD dataset includes geometry type, spatial reference and spatial range, etc.

[0032] Step 22, filtering the feature objects with empty geometry objects in the first RDD dataset to obtain a second RDD dataset.

[0033] In this step, the filter operator in Spark is used to filter out the features with empty geometry objects in the first RDD dataset to form the second RDD dataset. In the second RDD dataset, the geometry objects of all features are not empty, and the structure is the same as the first RDD dataset.

[0034] Step 23, performing aggregation operation on the second RDD dataset by taking the aggregation class and the feature objects in the second RDD dataset as parameters to obtain a third RDD dataset.

[0035] The third RDD dataset contains a final aggregation object.

[0036] The aggregation class StitchCombiner is used to aggregate features, and the aggregation class mainly includes an aggregation function combiner, a fusion data function mergeValue, an aggregation partition function mergeCombiner and a function getResult for obtaining aggregation results.

[0037] The specific steps of step 23 include steps 213-234: Step 231, the aggregation function creates an initial aggregation object for each partition according to the spatial reference and any feature object in each partition in the second RDD dataset.

[0038] In this step, the aggregation function is used to create a new aggregation object instance. For each partition, the aggregation function takes a spatial reference object and an element object in the partition as parameters, and returns a new aggregation object. Each partition creates a new aggregation object instance, that is, an initial aggregation object is created.

[0039] In step 232, the fusion data function performs boundary fusion on the element objects in each partition of the second RDD dataset, except for the element objects in the initial aggregation object, and the corresponding initial aggregation object, to obtain a partition aggregation object for each partition.

[0040] The fusion data function takes an aggregation object and an element object. The function is executed in a single partition, and performs boundary fusion on the polygon geometry of the elements in the aggregation object and the input element object. Boundary adjacent fusion is a polygon, and boundary non-adjacent fusion is not performed. Finally, a new aggregation object is returned. Among them, boundary adjacent refers to polygon boundary coincident and region overlapping polygons.

[0041] In this step, for a single partition, the initial aggregation object and the element objects in step 231 are boundary fused. Specifically, the aggregation logic of the fusion data function takes the initial aggregation object and an element object as parameters, performs boundary fusion on the polygon geometry of the two, returns an intermediate aggregation object (intermediate result), and then performs boundary fusion on the intermediate result and the next element object as parameters. The returned result is again boundary fused with the next element object, and so on, until all element objects in the current partition complete the boundary fusion operation, returning a partition aggregation object for the current partition. Each partition is boundary fused, and each partition obtains a partition aggregation object.

[0042] Here, the boundary fusion operation is implemented using an existing geometric fusion algorithm. For example, during fusion, a spatial index is first constructed, the geometric minimum boundary intersection is found, and then a topological edge graph is constructed for all found result geometries. The usage frequency of the edges is counted, and internal edges are removed according to the usage frequency, and a geometric contour is constructed.

[0043] In step 233, the aggregation partition function performs boundary fusion on the partition aggregation objects of different partitions to obtain a final aggregation object as the third RDD dataset.

[0044] In this step, the aggregation partition function takes two aggregation objects as parameters. The function performs boundary fusion on the aggregation result element geometries in the two aggregation objects when executed. Boundary adjacent fusion is a polygon, and boundary non-adjacent fusion is not performed. Finally, a new aggregation object is returned.

[0045] The aggregation logic of the aggregation partition function is to accept the partition aggregation object of the first partition and the partition aggregation object of the second partition as parameters, to perform boundary fusion on the aggregation result element geometries in the two aggregation objects, and to return a result (also an aggregation object). That is, the partition aggregation objects of two partitions are sequentially taken as parameters, boundary fusion is performed, a result is returned, the returned result is taken as a parameter, a secondary result is returned, the secondary result is taken as a parameter, and finally a result with only one aggregation object is returned. For example, the aggregation logic is as follows: Aggregation partition function (partition 1 aggregation object, partition 2 aggregation object) -> result A aggregation object; Aggregation partition function (partition 3 aggregation object, partition 4 aggregation object) -> result B aggregation object; Aggregation partition function (partition 5 aggregation object, partition 6 aggregation object) -> result C aggregation object; Aggregation partition function (partition 7 aggregation object, partition 8 aggregation object) -> result D aggregation object; Aggregation partition function (result A aggregation object, result B aggregation object) -> result E aggregation object; Aggregation partition function (result C aggregation object, result D aggregation object) -> result F aggregation object; Aggregation partition function (result E aggregation object, result F aggregation object) -> final result aggregation object.

[0046] Or Aggregation partition function (partition 1 aggregation object, partition 2 aggregation object) -> result A aggregation object; Aggregation partition function (partition 3 aggregation object, partition 4 aggregation object) -> result B aggregation object; Aggregation partition function (partition 5 aggregation object, partition 6 aggregation object) -> result C aggregation object; Aggregation partition function (result A aggregation object, result B aggregation object) -> result D aggregation object; Aggregation partition function (result C aggregation object, result D aggregation object) -> final result aggregation object.

[0047] Here, the boundary fusion operation is implemented by using an existing geometry fusion algorithm.

[0048] In step 234, an aggregation result function is used to obtain an aggregation result of the aggregation object, and an array of polygon geometry objects is returned. The aggregation result function is used to obtain a polygon geometry after aggregation, and an array of polygon geometry objects is returned.

[0049] Feature aggregation is based on the combineByKey operator in Spark, which processes the second RDD dataset, takes the custom aggregation class and feature object as parameters, uses the Spark distributed framework for calculation, and finally returns the third RDD dataset, which contains an aggregation object.

[0050] Step 24: Obtain the polygon geometry object array of the final aggregated object in the third RDD dataset to generate a target line feature object, thereby obtaining a fourth RDD dataset.

[0051] The fourth RDD dataset includes multiple rows of data, and each row of data contains a target line feature object.

[0052] Specifically, the Spark flatMap operator is used to obtain the polygon geometry array of the final aggregated object in the third RDD dataset. The geometric boundary coordinate data of each polygon geometry object is extracted to generate a target line geometry object. Target line feature objects are then generated based on the target line geometry objects. Flattening is then performed to convert all target line feature objects into multiple rows of data, each containing a target line feature object. A target line feature object consists of a target line geometry object and an empty attribute object.

[0053] Step 25, define the spatial reference and spatial range of the first RDD dataset as the spatial reference and spatial range of the fifth RDD dataset, define the geometry type of the fifth RDD dataset as a line type, and use each row of data of the fourth RDD dataset as the data of the fifth RDD dataset.

[0054] This step restructures the data, converting the fourth RDD into a custom FeatureRDD to facilitate data writeout and adaptation. Each row of the fifth RDD dataset contains a line feature, consisting of a target line geometry object and an empty attribute object. The fifth RDD metadata includes information such as the geometry type, spatial reference, and spatial extent of the spatial data.

[0055] Step 26: Write each row of data in the fifth RDD data set to the target database.

[0056] Use the foreachPartition function in Spark and the Hadoop related file writing class to write the fifth RDD dataset to a Shapefile in the local or HDFS. You can also write the fifth RDD dataset to a PostGIS spatial database based on the database connection information. The written data is the gap data detected by the spatial polygon data.

[0057] In this embodiment, for the problem that large-scale and large-volume spatial vector polygon data cannot be detected in a short time, the parallel processing can solve the problem of low efficiency and long time of traditional serial analysis. By using the Spark distributed computing framework, the spatial vector data can be analyzed and executed in parallel in a single machine or a multi-machine cluster, the computing resources can be fully utilized, the processing speed can be accelerated, and the large-scale and large-volume spatial vector data can be processed more easily.

[0058] In a feasible implementation, the above method is based on the Spark distributed framework and can be developed by using the Scala language. In addition, PySpark is a Spark interface library developed based on the Python language, which provides a set of functions for large-scale data processing. PySpark can use the Spark parallel technology to read the vector data in the local or HDFS and the vector data in the PostGIS database and implement the vector polygon gap detection. The above method can also use the Python language to write the calculation logic, thereby improving the universality and scalability.

[0059] As shown in Figure 2 Fig. 3, the third aspect of the embodiment of the present application provides a spatial vector polygon gap detection system based on the Spark framework, which comprises: A data reading module 31 is configured to read the polygon spatial vector data and create a first RDD data set according to the spatial vector data; wherein the first RDD data set comprises a plurality of partitions, each partition comprises a plurality of rows of data, each row of data contains an element object, and the geometric object in the first RDD data set is a polygon geometric object; A data filtering module 32 is configured to filter the element object with an empty geometric object in the first RDD data set to obtain a second RDD data set; A data aggregation module 33 is configured to aggregate the second RDD data set by taking the aggregation class and the element object in the second RDD data set as parameters to obtain a third RDD data set; wherein the third RDD data set contains a final aggregation object; A boundary extraction module 34 is configured to obtain the polygon geometric object array of the final aggregation object in the third RDD data set to generate a target line element object and obtain a fourth RDD data set; wherein the fourth RDD data set comprises a plurality of rows of data, and each row of data contains a target line element object; A data reconstruction module 35 is configured to take each row of data of the fourth RDD data set as each row of data of a fifth RDD data set, define the spatial reference and spatial range of the first RDD data set as the spatial reference and spatial range of the fifth RDD data set, and define the geometric type of the fifth RDD data set as a line type to obtain the fifth RDD data set; The data writing module 36 is configured to write each row of data in the fifth RDD dataset to the target database.

[0060] In one embodiment of the present application, the feature object in the first RDD dataset comprises a geometry object and an attribute object. The metadata information of the first RDD dataset comprises a geometry type, a spatial reference and a spatial extent.

[0061] In one embodiment of the present application, the feature object with the empty geometry object in the first RDD dataset is filtered to obtain the second RDD dataset, comprising: The feature object with the empty geometry object in the first RDD dataset is filtered by using the filter operator in Spark to obtain the second RDD dataset.

[0062] In one embodiment of the present application, the aggregation class comprises an aggregation function, a fusion data function, an aggregation partition function and an aggregation result acquisition function. The aggregation class and the feature object in the second RDD dataset are taken as parameters to perform the aggregation operation on the second RDD dataset to obtain the third RDD dataset, comprising: The aggregation function creates an initial aggregation object of each partition according to the spatial reference and any feature object in each partition in the second RDD dataset; The fusion data function performs the boundary fusion on the feature object in each partition of the second RDD dataset except the feature object in the initial aggregation object and the corresponding initial aggregation object to obtain a partition aggregation object of each partition; The aggregation partition function performs the boundary fusion on the partition aggregation objects of different partitions to obtain a final aggregation object as the third RDD dataset; The aggregation result acquisition function is configured to acquire the aggregation result of the final aggregation object and return an array of polygon geometry objects.

[0063] In one embodiment of the present application, the array of polygon geometry objects of the final aggregation object in the third RDD dataset is acquired to generate the target line feature object to obtain the fourth RDD dataset, comprising: The array of polygon geometry objects of the final aggregation object in the third RDD dataset is acquired by using the flatMap operator in Spark to generate the target line feature object to obtain the fourth RDD dataset.

[0064] In one embodiment of the present application, the array of polygon geometry objects of the final aggregation object in the third RDD dataset is acquired by using the flatMap operator in Spark to generate the target line feature object to obtain the fourth RDD dataset, comprising: The polygon geometry object array of the final aggregation object in the third RDD dataset is obtained through a flatMap operator in Spark, and the geometric boundary coordinate data of the polygon geometry object array is extracted to generate a target line geometry object, and then a target line feature object is generated according to the target line geometry object; wherein the target line feature object comprises the target line geometry object and an empty attribute object.

[0065] In an embodiment of the present application, each row of data of the fifth RDD dataset contains a line feature, and the line feature comprises a target line geometry object and an empty attribute object.

[0066] The fourth aspect of the embodiment of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned spatial vector polygon gap detection method based on the Spark framework provided by the embodiment of the present application when executing the program.

[0067] The fifth aspect of the embodiment of the present application further provides a computer readable storage medium having a computer program stored thereon, and the computer program is executable on the processor to implement the steps of the above-mentioned spatial vector polygon gap detection method based on the Spark framework provided by the embodiment of the present application.

[0068] The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage system located away from the aforementioned processor.

[0069] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware system.

[0070] The method provided by the embodiment of the present application can be applied to an electronic device. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. Herein, no limitation is made, and any electronic device that can implement the present application belongs to the protection scope of the present application.

[0071] For system / electronic device embodiments, because they are substantially similar to the method embodiments, the description is relatively simple, and the relevant parts are referred to the part of the method embodiments.

[0072] The present application is described in reference to the flowchart and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flow Figure 1 one or more flows and / or blocks Figure 1 a system with the function specified in one or more blocks.

[0073] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the flow Figure 1 one or more flows and / or blocks Figure 1 a system with the function specified in one or more blocks.

[0074] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the flow Figure 1 one or more flows and / or blocks Figure 1 a system with the function specified in one or more blocks.

[0075] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A method for detecting gaps in spatial vector polygons based on the Spark framework, characterized in that: The following steps are involved: Reading polygonal spatial vector data and creating a first RDD dataset based on the spatial vector data; wherein the first RDD dataset includes multiple partitions, each partition includes multiple rows of data, each row of data includes a feature object, and the geometric objects in the first RDD dataset are polygonal geometric objects; Filtering feature objects whose geometric objects are empty in the first RDD dataset to obtain a second RDD dataset; Performing an aggregation operation on the second RDD dataset using the aggregation class and the feature objects in the second RDD dataset as parameters to obtain a third RDD dataset; wherein the third RDD dataset contains a final aggregate object; Obtaining the polygon geometry object array of the final aggregated object in the third RDD dataset to generate a target line feature object, thereby obtaining a fourth RDD dataset; wherein the fourth RDD dataset includes multiple rows of data, and each row of data includes a target line feature object; Use each row of data in the fourth RDD dataset as each row of data in the fifth RDD dataset, define the spatial reference and spatial range of the first RDD dataset as the spatial reference and spatial range of the fifth RDD dataset, and define the geometry type of the fifth RDD dataset as the line type, to obtain the fifth RDD dataset; Write each row of data in the fifth RDD dataset to the target database.

2. The method according to claim 1, wherein The feature objects in the first RDD dataset include geometric objects and attribute objects; The metadata information of the first RDD data set includes: geometry type, spatial reference and spatial extent.

3. The method according to claim 1, wherein The filtering of feature objects whose geometric objects are empty in the first RDD dataset to obtain a second RDD dataset includes: The feature objects whose geometric objects are empty in the first RDD data set are filtered out by the filter operator in Spark to obtain a second RDD data set.

4. The method according to claim 2, wherein The aggregation class includes: aggregation function, data fusion function, aggregation partition function and aggregation result acquisition function; The step of performing an aggregation operation on the second RDD dataset using the aggregation class and the feature objects in the second RDD dataset as parameters to obtain a third RDD dataset includes: The aggregation function creates an initial aggregation object for each partition according to the spatial reference and any feature object in each partition in the second RDD dataset; The fusion data function performs boundary fusion on the feature objects other than the feature objects in the initial aggregation object in each partition of the second RDD data set and the corresponding initial aggregation object to obtain a partition aggregation object for each partition; The aggregation partition function performs boundary fusion on the partition aggregation objects of different partitions to obtain a final aggregation object as the third RDD data set; The Get Aggregation Result function is used to obtain the aggregation result of the final aggregation object and return an array of polygon geometry objects.

5. The method according to claim 1, wherein The step of obtaining the polygon geometry object array of the final aggregated object in the third RDD dataset to generate a target line feature object and obtain a fourth RDD dataset includes: The polygon geometry object array of the final aggregate object in the third RDD data set is obtained through the flatMap operator in Spark to generate a target line feature object, thereby obtaining a fourth RDD data set.

6. The method according to claim 5, wherein The method obtains the polygon geometry object array of the final aggregate object in the third RDD dataset by using the flatMap operator in Spark to generate a target line feature object, thereby obtaining a fourth RDD dataset, including: The polygon geometry object array of the final aggregate object in the third RDD data set is obtained through the flatMap operator in Spark, and the geometric boundary coordinate data of the polygon geometry object array is extracted to generate a target line geometry object, and then a target line feature object is generated based on the target line geometry object; wherein, the target line feature object includes the target line geometry object and an empty attribute object.

7. The method according to claim 6, wherein Each row of data in the fifth RDD dataset contains a line feature, and the line feature includes the target line geometry object and an empty attribute object.

8. A spatial vector polygon gap detection system based on Spark framework, characterized by: include: A data reading module, configured to read polygonal spatial vector data and create a first RDD dataset based on the spatial vector data; wherein the first RDD dataset includes a plurality of partitions, each partition includes a plurality of rows of data, each row of data includes a feature object, and the geometric objects in the first RDD dataset are polygonal geometric objects; A data filtering module, configured to filter feature objects whose geometric objects are empty in the first RDD dataset to obtain a second RDD dataset; a data aggregation module, configured to perform an aggregation operation on the second RDD dataset using the aggregation class and the feature objects in the second RDD dataset as parameters to obtain a third RDD dataset; wherein the third RDD dataset contains a final aggregate object; a boundary extraction module, configured to obtain an array of polygon geometry objects of the final aggregated object in the third RDD dataset to generate a target line feature object, thereby obtaining a fourth RDD dataset; wherein the fourth RDD dataset includes multiple rows of data, each row of data containing a target line feature object; a data reconstruction module, configured to use each row of data in the fourth RDD dataset as each row of data in a fifth RDD dataset, define the spatial reference and spatial range of the first RDD dataset as the spatial reference and spatial range of the fifth RDD dataset, define the geometry type of the fifth RDD dataset as a line type, and obtain a fifth RDD dataset; The data writing module is used to write each row of data in the fifth RDD data set to a target database.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the spatial vector polygon gap detection method based on the Spark framework is implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting gaps in spatial vector polygons based on the Spark framework according to any one of claims 1 to 7 is implemented.

Images