Vector data two-path space intersection method, device and system

Abstract

The invention discloses a vector data two-path space intersection method, device and system, and belongs to the technical field of geographic computation.The method comprises the steps that index blocks of different scales and an elastic distributed data set with index block identifiers as keys and a set of all vector data intersected with the index blocks as values are generated on the basis of a to-be-processed vector data set; inputting the elastic distributed data set into a trained index block calculation intensity prediction model to perform index block calculation intensity prediction, and obtaining the calculation intensity of each index block; based on the calculation intensity of each index block, mapping each index block to each task processing partition; performing two-path space intersection on vector data in the index block by utilizing each task processing partition; the index block calculation intensity prediction model is generated and trained by using a SparkML machine learning library based on a pre-acquired vector data set. According to the method, the two-path space intersection efficiency of the vector data can be improved, and meanwhile load balance in the parallel process is guaranteed.

Description

Technical Field

[0001] The invention relates to a two-way spatial intersection method, device and system for vector data, belonging to the technical field of geographic computing. Background Art

[0002] In recent years, with the widespread application of geographic information in the fields of transportation, land, water conservancy, etc., the scale of vector data has shown a trend of exponential growth, and has subsequently faced the problem of vector data processing at the TB or even PB level.

[0003] Two-way spatial vector intersection is the most basic and most commonly used spatial operation in vector data sets. However, as the scale of vector data continues to grow, the traditional serial method is too time-consuming and can no longer meet the needs of decision makers.

[0004] High-performance computing has been widely used in the field of geographic information. As a typical data-intensive application, two-way spatial vector intersection can be accelerated by high-performance parallel technology. However, as a kind of spatial data, vector data is usually irregular in spatial distribution and exhibits severe heterogeneity. Currently, open source frameworks that support parallel vector spatial intersection, such as Spatial Hadoop and Spatial Spark, have ignored the spatial heterogeneity of vector data, resulting in load imbalance and performance bottlenecks in the parallel process. Summary of the invention

[0005] The object of the present invention is to provide a method, device and system for two-way spatial intersection of vector data, which can improve the efficiency of two-way spatial intersection of vector data while ensuring load balancing in parallel processes.

[0006] In order to achieve the above object, the present invention provides the following technical solutions: In a first aspect, the present invention provides a two-way spatial intersection method for vector data, comprising: Get the vector dataset to be processed; Based on the vector data set to be processed, index blocks of different scales and a resilient distributed data set with an index block identifier as a key and a set of all vector data intersecting the index block as a value are generated; Inputting the elastic distributed data set into the trained index block computing strength prediction model to perform index block computing strength prediction, and obtaining the computing strength of each index block; Based on the computational intensity of each index block, each index block is mapped to each task processing partition; Using each task processing partition to perform a two-way spatial intersection on the vector data in each index block mapped thereto; The index block computing strength prediction model is generated and trained based on a pre-acquired vector data set using the SparkML machine learning library.

[0007] In combination with the first aspect, further, the vector data set includes vector data, and an object identifier and object geometry information corresponding to the vector data; wherein the object geometry information includes the structure, distribution and topological characteristics of the vector data.

[0008] In combination with the first aspect, further, based on the vector data set to be processed, generating index blocks of different scales includes: Based on the vector data set to be processed, the quadtree partitioning method is used to generate index blocks of different scales; The vector data intersecting with the index block is determined to fall into the index block.

[0009] In combination with the first aspect, further, based on the vector data set to be processed, generating an elastic distributed data set with the index block identifier as a key and the vector data set intersecting the index block as a value includes: Based on the vector dataset to be processed, generate a first dataset with the object identifier as a key and the object geometric information as a value; Based on the first data set, generate a second data set with the index block identifier as a key and the object identifier and object geometry information intersecting the index block as a value; Based on the second data set, generate a third data set with the index block identifier as a key and a set of vector data intersecting the index block as a value; The third data set is merged to generate a resilient distributed data set with the index block identifier as a key and a set of all vector data intersecting with the index block as a value.

[0010] In combination with the first aspect, further, based on the pre-acquired vector data set, using the SparkML machine learning library to generate and train an index block computing intensity prediction model includes: Generate training index blocks of different scales based on the pre-acquired vector dataset; Based on the training index block, generate an index block computing intensity prediction model using the SparkML machine learning library; Generate training samples based on the training index block, using the intersection time of the vector data in the training index block as a label and the structure, distribution and topological characteristics of the vector data as features; The training samples are input into the index block computing strength prediction model, and the SparkML machine learning library is used for training to obtain a trained index block computing strength prediction model.

[0011] In combination with the first aspect, further, based on the pre-acquired vector data set, generating training index blocks of different scales includes: Based on the pre-acquired vector data set, the quadtree partitioning method is used to generate training index blocks of different scales; The vector data intersecting with the training index block is determined to fall into the training index block.

[0012] In combination with the first aspect, further, based on the computing intensity of each index block, mapping each index block to each task processing partition includes: performing quadrilateral division on the index blocks whose computing intensity is greater than a preset threshold until the computing intensity of all index blocks is less than the preset threshold or the preset division termination condition is reached, and mapping each index block to each task processing partition so that the sum of the computing intensity of each index block mapped to each task processing partition is equal.

[0013] In combination with the first aspect, further, using each task processing partition to perform two-way spatial intersection on the vector data in each index block mapped thereto includes: using each task processing partition to construct an R-tree index for the vector data in each index block mapped thereto, and based on the R-tree index, performing two-way spatial intersection on the vector data in each index block.

[0014] In a second aspect, the present invention provides a two-way spatial intersection device for vector data, comprising: A data acquisition module, used to acquire a vector data set to be processed; A resilient distributed data set generation module, used for generating index blocks of different scales based on the vector data set to be processed, and a resilient distributed data set with an index block identifier as a key and a set of all vector data intersecting the index block as a value; A computing intensity prediction module is used to input the elastic distributed data set into a trained index block computing intensity prediction model to perform index block computing intensity prediction and obtain the computing intensity of each index block; An index block mapping module, used to map each index block to each task processing partition based on the computational intensity of each index block; An execution module, used for performing two-way spatial intersection on vector data in each index block mapped by each task processing partition; The index block computing strength prediction model is generated and trained based on a pre-acquired vector data set using the SparkML machine learning library.

[0015] In a third aspect, the present invention provides a computer system, comprising: Storage medium: used to store computer programs; Processor: used to execute the computer program to implement the steps of the two-way spatial intersection method of vector data described in any one of the first aspects.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the two-way spatial intersection method of vector data described in any one of the first aspects.

[0017] Fifthly, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the two-way spatial intersection method of vector data described in any one of the first aspects.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The vector data two-way space intersection method provided by the present invention is based on the SparkML machine learning library, obtains a trained index block computing intensity prediction model, predicts the computing intensity of each index block, and maps each index block to each task processing partition based on the computing intensity of each index block. Each task processing partition performs two-way space intersection on the vector data in the index block. The factors that affect the computing intensity of the two-way space intersection of vector data are taken into consideration, and parallel task complexity evaluation is realized. It can improve the efficiency of the two-way space intersection of vector data while ensuring load balancing in the parallel process.

[0019] The two-way spatial intersection method of vector data provided in this embodiment uses a quadtree partitioning method to generate training index blocks of different scales, which can improve the diversity of training samples and realize automatic generation of training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a two-way spatial intersection method of vector data provided by an embodiment of the present invention; Figure 2 The figure is a schematic diagram comparing the intersection execution time of the two-way spatial intersection method for vector data provided by an embodiment of the present invention and the open source cloud computing framework SpatialSpark under different index block partitioning parameters, wherein (1) represents the intersection execution time comparison under the index block partitioning parameter of 398×398, and (2) represents the intersection execution time comparison under the index block partitioning parameter of 798×798. DETAILED DESCRIPTION

[0021] The technical solution of the present application is further described in detail below in conjunction with specific implementation methods.

[0022] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and should not be construed as limitations on the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments may be combined with each other.

[0023] Embodiment 1: Figure 1 is a flow chart of a two-way spatial intersection method of vector data provided in this embodiment. This flow chart only shows the logical sequence of the method of this embodiment. Different methods can be used without conflict. Figure 1 The steps shown or described are accomplished in the order shown.

[0024] The vector data two-way spatial intersection method provided in this embodiment can be applied to a terminal and can be executed by a vector data two-way spatial intersection device, which can be implemented by software and / or hardware, and can be integrated in a terminal, for example: any tablet computer or computer device with communication function.

[0025] See also Figure 1 The vector data two-way spatial intersection method of this embodiment specifically includes the following steps: Step 1: Get the vector data set to be processed; In this embodiment, the vector data set includes vector data, and an object identifier and object geometry information corresponding to the vector data; wherein the object geometry information includes the structure, distribution, and topological characteristics of the vector data.

[0026] Step 2: Based on the vector data set to be processed, generate index blocks of different scales and a resilient distributed data set with the index block identifier as the key and the set of all vector data intersecting the index block as the value; In this embodiment, generating index blocks of different scales based on the vector data set to be processed specifically includes: generating index blocks of different scales based on the vector data set to be processed by using a quadtree partitioning method.

[0027] The vector data set to be processed is divided by using the quadtree partitioning method to generate index blocks of different scales, and the vector data intersecting with the index block is judged to fall into the index block.

[0028] In this embodiment, based on the vector data set to be processed, generating an elastic distributed data set with an index block identifier as a key and a vector data set intersecting with the index block as a value specifically includes the following steps: Step ①: Based on the vector data set to be processed, generate a first data set with the object identifier as the key and the object geometric information as the value; In this embodiment, based on the vector datasets R and S to be processed, a first dataset R-PairRDD is generated with the object identifier OID as the key and the object geometric information GEO as the value.<OID,GEO> and S-PairRDD<OID,GEO> .

[0029] Step ②: Based on the first data set, generate a second data set with the index block identifier as the key and the object identifier and object geometry information intersecting with the index block as the value; In this embodiment, the block information is initialized based on the index block partitioning parameters, and the intersection relationship between the vector data and the index block is determined through the Spark flatMap operation.<OID,GEO> and S-PairRDD<OID,GEO> , respectively generate the second data set R-PairRDD with the index block identifier GID as the key, the object identifier OID intersecting with the index block and the object geometry information GEO as the value<GID,(OID,GEO)> and S-PairRDD<GID,(OID,GEO)> .

[0030] Step ③: Based on the second data set, generate a third data set with the index block identifier as the key and the set of vector data intersecting the index block as the value; In this embodiment, all vector data intersecting with the same index block are collected through the Spark GroupByKey operation, based on the second data set R-PairRDD<GID,(OID,GEO)> and S-PairRDD<GID,(OID,GEO)> , respectively generate a third data set R-PairRDD with the index block identifier GID as the key and the set of vector data intersecting the index block as the value<GID,List(OID,GEO)> and S-PairRDD<GID, List (OID,GEO)> , where List(OID,GEO) represents the collection of vector data that intersects the index block.

[0031] Step ④: Merge the third data set to generate a resilient distributed data set with the index block identifier as the key and the set of all vector data intersecting the index block as the value.

[0032] In this embodiment, the third data set R-PairRDD is processed by Spark Join operation.<GID,List(OID,GEO)> and S-PairRDD<GID, List (OID,GEO)> Merge based on index blocks and generate a resilient distributed dataset PairRDD with the index block identifier GID as the key and the set of all vector data intersecting the index block as the value.<GID, (R-List, S-List)> , where R-List and S-List represent the sets of vector data in R and S that intersect with the index block.<GID, (R-List, S-List)> Each record in corresponds to an index block and a set of vector data in R and S that intersects with the index block.

[0033] The two-way spatial intersection method of vector data provided in this embodiment uses the Spark framework to read the vector data as a resilient distributed dataset (RDD) of key-value pair type, wherein the key is the unique ID of the vector data, that is, the object identifier OID corresponding to the vector data, and the value is the object geometry information GEO corresponding to the vector data. After the intersection judgment with the index block, the key of the RDD is converted to the unique ID of the index block, that is, the index block identifier GID, and the value is converted to the object identifier OID and the object geometry information GEO that intersect with the index block. The vectors that fall into the same index block, that is, the vectors that intersect with the same index block, are collected through the Spark GroupByKey operation, and the sets of vector data that intersect with the index block in R and S are merged based on the index block through the Spark Join operation, which can help to directly perform feature extraction and computational intensity prediction for the index block.

[0034] Step 3: Input the elastic distributed data set into the trained index block computing strength prediction model to predict the index block computing strength and obtain the computing strength of each index block; In this embodiment, the index block computing strength prediction model is generated and trained based on the pre-acquired vector data set using the SparkML machine learning library.

[0035] Based on the pre-acquired vector data set, the SparkML machine learning library is used to generate and train the index block computing intensity prediction model, which specifically includes the following steps: Step 1: Generate training index blocks of different scales based on the pre-acquired vector dataset; In this embodiment, generating training index blocks of different scales based on the pre-acquired vector data set specifically includes: generating training index blocks of different scales based on the pre-acquired vector data set using a quadtree partitioning method.

[0036] The pre-acquired vector data set is divided by using a quadtree partitioning method to generate training index blocks of different scales, and the vector data intersecting with the training index block is determined to fall into the training index block.

[0037] Step 2: Based on the training index block, use the SparkML machine learning library to generate an index block computing intensity prediction model; In this embodiment, based on the training index block, the random forest model provided by the SparkML machine learning library is used to generate an index block computing strength prediction model, and the serialization interface provided by the SparkML machine learning library is used to store the index block computing strength prediction model on the distributed file system (Hadoop Distributed File System, HDFS).

[0038] Step 3: Generate training samples based on the training index block, taking the intersection time of the vector data in the training index block as the label and the structure, distribution and topological characteristics of the vector data as the features; In this embodiment, the intersection time of the vector data in the training index block is calculated as the computing intensity of the training index block, and feature extraction is performed on the training index block, including feature extraction of the structure, distribution and topological characteristics of the vector data in the training index block. The intersection time of the vector data in the training index block is used as a label, and the structure, distribution and topological characteristics of the vector data are used as features to generate training samples.

[0039] The two-way spatial intersection method of vector data provided in this embodiment uses a quadtree partitioning method to generate training index blocks of different scales, which can improve the diversity of training samples. The computational intensity characteristics of the two-way spatial intersection of vector data are mined according to the expert knowledge base, and the intersection time of the vector data in the training index block is used as the label of the training sample, so that the training sample is automatically generated based on the machine without manual intervention. Different hardware has different computing time, which is proportionally enlarged or reduced to reduce the impact on load balancing.

[0040] Step 4: Input the training samples into the index block computing intensity prediction model, use the SparkML machine learning library for training, and obtain the trained index block computing intensity prediction model.

[0041] In this embodiment, the training samples are input into the index block computing strength prediction model, the random forest model provided by the SparkML machine learning library is used to train the index block computing strength prediction model, the trained index block computing strength prediction model is obtained, and the serialization interface provided by the SparkML machine learning library is used to store the trained index block computing strength prediction model on the distributed storage system HDFS, so that the model can be read in parallel during subsequent distributed computing.

[0042] The vector data two-way spatial intersection method provided in this embodiment uses the SparkML machine learning library to train, store, and apply the index block calculation intensity prediction model to achieve parallel task complexity evaluation. The factors that affect the calculation intensity of the two-way spatial intersection of vector data are considered, rather than just using the number of vectors as an indicator for evaluating the calculation intensity. Steps 1 to 4 are all offline training, which is completed in advance using the model and interface provided by the SparkML machine learning library, and the model training is not included in the parallel execution process of the actual application.

[0043] In this embodiment, the trained index block computing strength prediction model stored in HDFS is read, and an elastic distributed dataset PairRDD with the index block identifier GID as the key and the set of all vector data intersecting the index block as the value is constructed.<GID, (R-List, S-List)> The trained index block computing strength prediction model is input to perform index block computing strength prediction to obtain the computing strength CI of each index block.

[0044] Step 4: Map each index block to each task processing partition based on the computational intensity of each index block; In this embodiment, based on the computational strength of each index block, mapping each index block to each task processing partition specifically includes: performing quadrilateral division on the index blocks whose computational strength is greater than a preset threshold until the computational strength of all index blocks is less than the preset threshold or the preset division termination condition is reached, and mapping each index block to each task processing partition so that the sum of the computational strengths of the index blocks mapped to each task processing partition is equal.

[0045] In this embodiment, for index blocks with computing intensity greater than a preset threshold, a custom splitWithCI interface is used to perform quadripartition until the computing intensity of all index blocks is less than the preset threshold or the preset partition termination condition is reached, and a resilient distributed dataset PairRDD is generated with the index block identifier GID as the key, the set of all vector data intersecting the index block, and the computing intensity CI of the index block as the value.<GID, ((R-List, S-List),CI)> , and use the Cache interface for caching.

[0046] Custom splitWithCI interface with PairRDD<GID, (R-List, S-List)> , the trained index block calculation intensity prediction model, the maximum number of quad partitions as input, and PairRDD<GID, ((R-List, S-List),CI)> As output, PairRDD can be<GID, (R-List, S-List)> Input the trained index block computing strength prediction model to predict the computing strength of the index block, and obtain the computing strength of the index block. When the computing strength of the index block is greater than the preset threshold and the number of quadripartitions is less than the maximum number of quadripartitions, quadripartition the index block until the computing strength of all index blocks is less than the preset threshold or the number of quadripartitions reaches the maximum number of quadripartitions, and output PairRDD<GID,((R-List, S-List),CI)> .

[0047] In this embodiment, the PairRDD cached based on the Cache interface<GID, ((R-List, S-List),CI)> , use the custom determineBoundsWithCI interface to map each index block to each task processing partition, so that the sum of the computational intensities of each index block mapped by each task processing partition is equal, and output the index block partition boundary.

[0048] Custom determineBoundsWithCI interface with PairRDD<GID, ((R-List, S-List),CI)> , the number of partitions is taken as input, and the index block partition boundary is taken as output. It can map each index block to each task processing partition according to the computing intensity CI of each index block and the number of partitions, so that the sum of the computing intensity of each index block mapped by each task processing partition is equal, and can calculate the index block partition boundary according to the computing intensity CI of each index block and the number of partitions.

[0049] In this embodiment, based on the index block partition boundary, a custom CIPredictPartitioner partitioner is used to partition PairRDD.<GID, ((R-List, S-List),CI)> Perform repartitioning.

[0050] The custom CIPredictPartitioner partitioner can partition PairRDD according to the index block partition boundary.<GID, ((R-List, S-List),CI)> Perform repartitioning.

[0051] The two-way spatial intersection method of vector data provided in this embodiment divides the coarse-grained task into fine-grained tasks through the customized splitWithCI interface, and constructs a load-balanced Spark task processing partition through the customized CIPredictPartitioner interface.

[0052] Step 5: Use each task processing partition to perform a two-way spatial intersection on the vector data in each index block mapped to it.

[0053] In this embodiment, using each task processing partition to perform two-way spatial intersection on the vector data in each index block mapped thereto specifically includes: using each task processing partition to construct an R-tree index on the vector data in each index block mapped thereto, and based on the R-tree index, performing two-way spatial intersection on the vector data in each index block.

[0054] In this embodiment, each task processing partition is used to process PairRDD<GID, ((R-List, S-List),CI)> Build an R-tree index from the vector data in the R-List or S-List in the image, and perform an intersection operation to generate a resilient distributed vector dataset PairRDD<R-OID,S-OID> , where R-OID and S-OID represent the object identifiers of the intersecting vector data in R and S.

[0055] The two-way spatial intersection method of vector data provided in this embodiment uses task processing partitions to construct an R-tree index for the vector data in each index block, which can further improve the intersection speed.

[0056] like Figure 2 As shown, under different index block partitioning parameters, the two-way spatial partitioning method for vector data provided in this embodiment is compared with the open source cloud computing framework SpatialSpark.

[0057] Depend on Figure 2 As can be seen from (1), when the index block partition parameter is 398×398, the two-way spatial partitioning method for vector data provided in this embodiment significantly shortens the intersection execution time on 12-core, 16-core, 24-core, and 48-core task processing partitions compared to the open source cloud computing framework SpatialSpark. Figure 2 As can be seen from (2), when the index block partition parameter is 798×798, the two-way spatial partitioning method for vector data provided in this embodiment significantly shortens the intersection execution time on 12-core, 16-core, 24-core, and 48-core task processing partitions compared to the open source cloud computing framework SpatialSpark.

[0058] Depend on Figure 2 It can be seen that the two-way spatial intersection method of vector data provided in this embodiment can increase the intersection speed by up to 80% compared with the open source cloud computing framework SpatialSpark, which can effectively save the intersection execution time.

[0059] Embodiment 2: This embodiment provides a two-way spatial intersection device for vector data, including: A data acquisition module, used to acquire a vector data set to be processed; A resilient distributed data set generation module, used for generating index blocks of different scales based on the vector data set to be processed, and a resilient distributed data set with an index block identifier as a key and a set of all vector data intersecting the index block as a value; A computing intensity prediction module is used to input the elastic distributed data set into the trained index block computing intensity prediction model to perform index block computing intensity prediction and obtain the computing intensity of each index block; An index block mapping module, used to map each index block to each task processing partition based on the computational intensity of each index block; An execution module, used for performing two-way spatial intersection on vector data in each index block mapped by each task processing partition; Among them, the index block computing intensity prediction model is generated and trained based on the pre-acquired vector dataset using the SparkML machine learning library.

[0060] The vector data two-way spatial intersection device provided in the embodiment of the present application can execute the vector data two-way spatial intersection method provided in Example 1 of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0061] Embodiment 3: This embodiment provides a computer system, including: Storage medium: used to store computer programs; Processor: used to execute computer programs to implement the steps of the two-way spatial intersection method of vector data in Example 1.

[0062] Embodiment 4: This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the two-way spatial intersection method of vector data in Embodiment 1 are implemented.

[0063] Embodiment 5: This embodiment provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the two-way spatial intersection method of vector data in Embodiment 1 are implemented.

[0064] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0065] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0066] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0068] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A two-way spatial intersection method for vector data, characterized in that: include: Get the vector dataset to be processed; Based on the vector data set to be processed, index blocks of different scales and a resilient distributed data set with an index block identifier as a key and a set of all vector data intersecting the index block as a value are generated; Inputting the elastic distributed data set into the trained index block computing strength prediction model to perform index block computing strength prediction, and obtaining the computing strength of each index block; Based on the computational intensity of each index block, each index block is mapped to each task processing partition; Using each task processing partition to perform a two-way spatial intersection on the vector data in each index block mapped thereto; The index block computing strength prediction model is generated and trained based on a pre-acquired vector data set using the SparkML machine learning library.

2. The two-way spatial intersection method of vector data according to claim 1, characterized in that: The vector data set includes vector data, and an object identifier and object geometry information corresponding to the vector data; wherein the object geometry information includes the structure, distribution and topological characteristics of the vector data.

3. The two-way spatial intersection method of vector data according to claim 1, characterized in that: Based on the vector dataset to be processed, index blocks of different scales are generated, including: Based on the vector data set to be processed, the quadtree partitioning method is used to generate index blocks of different scales; The vector data intersecting with the index block is determined to fall into the index block.

4. The two-way spatial intersection method of vector data according to claim 1, characterized in that: Based on the vector dataset to be processed, generating a resilient distributed dataset with an index block identifier as a key and a vector dataset intersecting the index block as a value includes: Based on the vector dataset to be processed, generate a first dataset with the object identifier as a key and the object geometric information as a value; Based on the first data set, generate a second data set with the index block identifier as a key and the object identifier and object geometry information intersecting the index block as a value; Based on the second data set, generate a third data set with the index block identifier as a key and a set of vector data intersecting the index block as a value; The third data set is merged to generate a resilient distributed data set with the index block identifier as a key and a set of all vector data intersecting with the index block as a value.

5. The two-way spatial intersection method of vector data according to claim 1, characterized in that: Based on the pre-acquired vector dataset, the SparkML machine learning library is used to generate and train the index block computing intensity prediction model, including: Generate training index blocks of different scales based on the pre-acquired vector dataset; Based on the training index block, generate an index block computing intensity prediction model using the SparkML machine learning library; Generate training samples based on the training index block, using the intersection time of the vector data in the training index block as a label and the structure, distribution and topological characteristics of the vector data as features; The training samples are input into the index block computing strength prediction model, and the SparkML machine learning library is used for training to obtain a trained index block computing strength prediction model.

6. The two-way spatial intersection method of vector data according to claim 5, characterized in that: Based on the pre-acquired vector dataset, training index blocks of different scales are generated, including: Based on the pre-acquired vector data set, the quadtree partitioning method is used to generate training index blocks of different scales; The vector data intersecting with the training index block is determined to fall into the training index block.

7. The vector data two-way spatial intersection method according to claim 1, characterized in that: Based on the computational intensity of each index block, mapping each index block to each task processing partition includes: performing quadrilateral division on the index blocks whose computational intensity is greater than a preset threshold until the computational intensity of all index blocks is less than the preset threshold or the preset division termination condition is reached, and mapping each index block to each task processing partition so that the sum of the computational intensities of the index blocks mapped to each task processing partition is equal.

8. The two-way space partitioning method for vector data according to claim 1, characterized in that: Using each task processing partition to perform two-way spatial intersection on the vector data in each index block mapped thereto includes: using each task processing partition to construct an R-tree index on the vector data in each index block mapped thereto, and based on the R-tree index, performing two-way spatial intersection on the vector data in each index block.

9. A two-way spatial intersection device for vector data, characterized in that: include: A data acquisition module, used to acquire a vector data set to be processed; A resilient distributed data set generation module, used for generating index blocks of different scales based on the vector data set to be processed, and a resilient distributed data set with an index block identifier as a key and a set of all vector data intersecting the index block as a value; A computing intensity prediction module is used to input the elastic distributed data set into a trained index block computing intensity prediction model to perform index block computing intensity prediction and obtain the computing intensity of each index block; An index block mapping module, used to map each index block to each task processing partition based on the computational intensity of each index block; An execution module, used for performing two-way spatial intersection on vector data in each index block mapped by each task processing partition; The index block computing strength prediction model is generated and trained based on a pre-acquired vector data set using the SparkML machine learning library.

10. A computer system, characterized in that: include: Storage medium: used to store computer programs; Processor: used to execute the computer program to implement the steps of the two-way spatial intersection method of vector data as described in any one of claims 1 to 8.

Images