A distributed processing method and device for spatial big data

Abstract

The present invention provides a distributed processing method and device for spatial big data. The method comprises acquiring spatial data, which includes vector data; performing gridded geographic partitioning on the spatial data according to a pre-set spatial indexing method that matches the data distribution characteristics of the spatial data, and generating a spatial index file, wherein the spatial index file includes at least a plurality of feature object files; and storing the plurality of feature object files in the spatial index file in a distributed file system (HDFS). When spatial data calculations are required, computing nodes in Spark are called in parallel, so that each computing node obtains a plurality of target feature object files with the same feature object file name from various storage nodes in the distributed file system (HDFS), and performs data processing using the spatial data contained in the plurality of target feature object files. The present invention can improve the processing efficiency of spatial data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a distributed processing method and device for spatial big data. Background Art

[0002] As the amount of spatial data obtained using traditional surveying and mapping methods, new surveying and mapping methods, and the Internet increases, and these massive spatial data also have different data structures and storage methods, how to perform high-performance distributed storage, management, and analytical calculations on massive spatial data is an urgent problem that needs to be solved.

[0003] Existing data analysis and calculation methods, due to the limitations of the IT technology framework and the technical requirements of big data for distributed storage and computing, cannot process massive spatial data well. It takes a certain amount of time to build a geographic partition spatial index during the calculation process, which increases the overall calculation time and is not efficient enough in processing massive spatial data. Summary of the Invention

[0004] In view of this, the present invention provides a distributed processing method and device for spatial big data, in order to improve the processing efficiency of spatial data.

[0005] A first aspect of the present invention discloses a distributed processing method for spatial big data, the method comprising:

[0006] Acquiring spatial data, wherein the spatial data includes vector data;

[0007] Performing gridded geographical partitioning on the spatial data according to a preset spatial indexing method that matches the data distribution characteristics of the spatial data to generate a spatial index file, wherein the spatial index file includes at least a plurality of feature object files, and the data distribution characteristics of the spatial data are uniform distribution, or the data distribution characteristics of the spatial data are uneven distribution and the spatial data is in a clustered state;

[0008] The multiple feature object files in the spatial index file are stored in a distributed file system HDFS.

[0009] Optionally, each of the feature object files includes at least a feature object file name and spatial data corresponding to a geographical partition after gridding the geographical partitions. After storing the multiple feature object files in a distributed file system HDFS, the method further includes:

[0010] The computing nodes in Spark are called in parallel to execute the corresponding steps. The corresponding steps executed by each computing node are:

[0011] A plurality of target feature object files having the same feature object file name are obtained from each storage node of the distributed file system HDFS, and data processing is performed using the spatial data contained in the plurality of target feature object files.

[0012] Optionally, if the data distribution characteristic of the spatial data is uniform distribution, gridding the spatial data according to a preset spatial indexing method that matches the data distribution characteristic of the spatial data to generate a spatial index file includes:

[0013] If the data distribution characteristic of the spatial data is uniform distribution, determining that the spatial indexing method matching the spatial data is a uniform grid spatial indexing method;

[0014] Based on the uniform grid spatial indexing method, the spatial data is subjected to gridded geographical partitioning to generate a spatial index file.

[0015] Optionally, the gridding geographical partitioning of the spatial data based on the uniform grid spatial indexing method to generate a spatial index file includes:

[0016] Based on the uniform grid spatial index method, the spatial data is gridded and geographically partitioned to obtain a DSF feature dataset, wherein the DSF feature dataset includes multiple geographical partitions and spatial data corresponding to each geographical partition;

[0017] For each of the geographical partitions, generating a feature object file corresponding to the geographical partition using the spatial data corresponding to the geographical partition;

[0018] A spatial index file is generated based on all the feature object files.

[0019] Optionally, if the data distribution characteristics of the spatial data are uneven distribution and the spatial data is in a clustered state, the spatial data is subjected to gridded geographical partitioning according to a preset spatial indexing method that matches the data distribution method of the spatial data to generate a spatial index file, including:

[0020] If the data distribution characteristic of the spatial data is uneven distribution and the spatial data is in a clustered state, determining that the spatial indexing method that matches the spatial data is a quadtree grid spatial indexing method;

[0021] Based on the quadtree grid spatial indexing method, the spatial data is subjected to gridded geographical partitioning to generate a spatial index file.

[0022] Optionally, the gridding geographical partitioning of the spatial data based on the quadtree grid spatial indexing method to generate a spatial index file includes:

[0023] Based on the quadtree grid spatial indexing method, the spatial data is gridded and geographically partitioned to obtain a DSF feature dataset, a plurality of geographical partitions of the DSF feature dataset and spatial data corresponding to each geographical partition;

[0024] For each of the geographical partitions, generating a feature object file corresponding to the geographical partition using the geographical partition and the corresponding spatial data;

[0025] A spatial index file is generated based on all the feature object files.

[0026] A second aspect of the present invention discloses a distributed processing device for spatial big data, the device comprising:

[0027] an acquisition unit, configured to acquire spatial data, wherein the spatial data includes vector data;

[0028] a first geographic partitioning unit, configured to perform gridded geographic partitioning on the spatial data according to a preset spatial indexing method that matches the data distribution characteristics of the spatial data, to obtain a spatial index file, wherein the spatial index file includes at least a plurality of feature object files, and the data distribution characteristics of the spatial data are uniform distribution, or the data distribution characteristics of the spatial data are non-uniform distribution and the spatial data is in a clustered state;

[0029] The storage unit is used to store the multiple feature object files in the spatial index file in a distributed file system HDFS.

[0030] Optionally, each of the feature object files includes at least a feature object file name and spatial data corresponding to a geographical partition after gridding, and the device further includes:

[0031] The execution unit is used to call the computing nodes in Spark in parallel to execute the corresponding steps. The corresponding steps executed by each computing node are:

[0032] A plurality of target feature object files having the same feature object file name are obtained from each storage node of the distributed file system HDFS, and data processing is performed using the spatial data contained in the plurality of target feature object files.

[0033] The third aspect of the present invention shows an electronic device, which includes a processor and a memory, wherein the memory is used to store program code and data for distributed processing of spatial big data, and the processor is used to call the program instructions in the memory to execute a distributed processing method of spatial big data as shown in the first aspect of the present invention.

[0034] The fourth aspect of the present invention shows a storage medium, which includes a storage program, wherein when the program is running, the device where the storage medium is located is controlled to execute a distributed processing method of spatial big data as shown in the first aspect of the present invention.

[0035] The present invention provides a distributed processing method and device for spatial big data. A corresponding spatial indexing method is set for spatial data with different data distribution characteristics, wherein the data distribution characteristic of the spatial data is uniform distribution, or the data distribution characteristic of the spatial data is uneven distribution and the spatial data is in a clustered state. After acquiring the spatial data, the acquired spatial data is gridded and geographically partitioned according to a preset spatial indexing method that matches the data distribution characteristic of the spatial data, and a spatial index file containing multiple feature object files is generated. The multiple feature files in the spatial index file are stored in a distributed file system HDFS, so that when spatial data calculation is required, the computing nodes in Spark are called in parallel, so that each computing node obtains multiple target feature object files with the same feature object file name from each storage node of the distributed file system HDFS, and uses the spatial data contained in the multiple target feature object files for data processing. The technical solution provided by the present invention has already gridded the spatial data before calculating it, so when calculating the spatial data, there is no need to geographically partition the spatial data, which reduces the time consumption of the calculation process. Moreover, when calculating the spatial data, by calling the various computing nodes in Spark to perform parallel calculations, the time consumption of the calculation process is further reduced, thereby greatly improving the processing efficiency of the spatial data. In other words, the present invention can realize the rapid distributed storage, management and analysis calculation of billions of spatial data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0037] Figure 1 A schematic diagram of a flow chart of a distributed processing method for spatial big data provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a uniform grid spatial indexing method provided by an embodiment of the present invention;

[0039] Figure 3A schematic diagram of a quadtree grid spatial indexing method provided by an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of the structure of a distributed processing device for spatial big data provided by an embodiment of the present invention;

[0041] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0044] It should be noted that the concepts of "first" and "second" mentioned in the disclosure of the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0045] It should be noted that the modifications of "one" and "multiple" mentioned in the disclosure of the present invention are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0046] DSF: Distributed Spatial File Engine, DistributedSpatialFile, is a new type of spatial data storage format. The data format obtained by gridding spatial data into geographical partitions is DSF. Distributed storage and calculation using spatial data in this format can more efficiently solve the distributed computing problem of massive data.

[0047] See also Figure 1 , which shows a flow chart of a distributed processing method for spatial big data provided by an embodiment of the present invention. The data processing method specifically includes the following steps:

[0048] S101: Acquire spatial data, which includes vector data.

[0049] In the embodiment of the present application, the acquired spatial data may be massive spatial data, and the acquired spatial data may be vector data.

[0050] It should be noted that the spatial data can be massive spatial data, or TB / PB-level multi-source heterogeneous spatial data, and the data format of the vector data can be a database type data format, GeoJSON data format, or CSV data format.

[0051] S102: Gridding the spatial data into geographical partitions according to a preset spatial indexing method that matches the data distribution characteristics of the spatial data, and generating a spatial index file, wherein the spatial index file includes at least a plurality of feature object files.

[0052] In step S102, by analyzing the spatial data, it can be determined that the data distribution characteristics of the spatial data can be uniform distribution, or uneven distribution and presenting a clustered state. For spatial data with uniform data distribution characteristics, the uniform grid spatial index method can be pre-set to the spatial index method corresponding to the uniform distribution characteristics of the spatial data; for spatial data with uneven data distribution characteristics and presenting a clustered state, the quadtree grid spatial index method can be pre-set to the spatial index method corresponding to the uneven distribution characteristics of the spatial data and presenting a clustered state.

[0053] It should be noted that the principle of uniform grid spatial indexing is as follows Figure 2 As shown, specifically, the spatial data can be divided into grids and numbered. It can be determined that there are three spatial objects in a certain spatial range, and the ids of these spatial objects are 8, 11, and 21. At this time, the spatial objects are stored in the grid range. For example, for grid No. 2, the spatial object with id = 5 is stored, and for grid No. 75, the spatial objects with id = 5 and 23 are stored. If you want to know which spatial objects are in a certain spatial range, see Figure 2 The red box (spatial range) in the figure can be used to determine the intersection of grid 75 and grid 76 according to the spatial range indicated by the red box, and then the IDs corresponding to the corresponding spatial objects can be queried according to the corresponding grids, that is, the queried IDs are id8 and id11 respectively.

[0054] It should be noted that the quadtree grid spatial indexing method recursively divides the spatial data into four parts until the self-set termination condition is reached (for example, the number of elements associated with each node does not exceed 3. If it exceeds 3, it will be divided into four parts again), and finally a hierarchical quadtree is formed. The rectangle with a number in the figure is the MBR of each element. Each leaf node stores the element identifier list associated with the area and the geographical scope of the area. Non-leaf nodes only store the geographical scope of the area. Figure 3 shown.

[0055] During the specific execution of step S102, after obtaining the spatial data, the data distribution characteristics of the obtained spatial data can be determined. If the data distribution characteristics of the obtained spatial data are determined to be uniform, the spatial index method that matches the spatial data is determined to be a uniform grid spatial index method. The uniform grid spatial index method is then used to perform gridded geographic partitioning on the spatial data to obtain a DSF feature dataset, index information, and summary information. For each geographic partition in the DSF feature dataset, the spatial data corresponding to the geographic partition is used to generate a feature object file corresponding to the geographic partition. The corresponding spatial index file is then generated based on all the feature object files, index information, and summary information. The DSF feature dataset is also referred to as a geographic partition feature dataset.

[0056] If it is determined that the data distribution characteristics of the acquired spatial data are unevenly distributed and present an aggregated state, it is determined that the spatial index method matching the spatial data is the quadtree grid spatial index method, and then the quadtree grid spatial index method is used to perform gridded geographical partitioning on the spatial data to obtain a DSF feature dataset, index information, and summary information; for each geographical partition in the DSF feature dataset, the spatial data corresponding to the geographical partition is used to generate a feature object file corresponding to the geographical partition; and then a corresponding spatial index file is generated based on all the feature object files, index information, and summary information.

[0057] It should be noted that the DSF feature dataset includes multiple geographic partitions and the spatial data corresponding to each geographic partition; the index information includes at least the geographic partition dataset type, the projection information of each geographic partition in the geographic partition data, and the geographic range; the summary information includes at least the number of geometric objects and the latest editing time, etc.; each feature object file includes the feature object file name corresponding to the feature object file and the spatial data corresponding to a geographic partition after gridded geographic partitioning.

[0058] In an embodiment of the present application, in addition to gridding the spatial data into geographic partitions using the above two methods to obtain a DSF feature dataset, a corresponding DSF feature dataset can also be constructed based on the existing geographic partition index. For example, when performing distributed operations such as overlay analysis and feature connection between two or more feature datasets, it is often necessary to keep the partition indexes of these two or more datasets consistent. Therefore, an index can be first constructed for one of the datasets with a larger spatial range, and then an index can be constructed for the other dataset based on this index.

[0059] It should be noted that the spatial data is gridded and geographically partitioned using a uniform grid spatial index method or a quadtree grid spatial index method, and the file format of the DSF feature dataset is a distributed spatial file engine (DistributedSpatialFile, DSF).

[0060] S103: Storing the multiple feature object files in the spatial index file in the distributed file system HDFS.

[0061] In the specific process of executing step S103, after the spatial data is gridded and geographically partitioned according to a pre-set spatial indexing method that matches the data distribution characteristics of the spatial data and the corresponding spatial index file is generated, the multiple feature object files in the spatial index file can be stored in the distributed file system (Hadoop Distributed File System, HDFS) in the form of a distributed spatial file engine DSF.

[0062] It should be noted that when the distributed file system HDFS performs distributed storage on multiple feature object files, it can use the Parquet file format to perform distributed storage on multiple feature object files. Parquet is an analysis-oriented, general-purpose columnar storage format that is compatible with the Spark data processing framework. This columnar storage method can be used to read files more efficiently.

[0063] S104: Call each computing node in Spark in parallel to execute corresponding steps. The corresponding steps executed by each computing node are: obtaining multiple target feature object files with the same feature object file name from each storage node of the distributed file system HDFS, and using the spatial data contained in the multiple target feature object files to perform data processing.

[0064] In the specific process of executing step S104, after storing multiple feature object files in the spatial index file to the distributed file system HDFS, the computing nodes in Spark can be called in parallel. For each computing node, the computing node can obtain multiple target feature object files with the same feature object file name from each storage node of the distributed file system HDFS, and use the spatial data contained in the multiple target feature object files to perform data processing.

[0065] It should be noted that some computing nodes in Spark can be called in parallel, or each computing node in Spark can be called in parallel. Specifically, multiple computing nodes in Spark can be called in parallel according to actual applications, and the embodiments of this application are not limited thereto.

[0066] The present invention provides a distributed processing method for spatial big data. A corresponding spatial indexing method is set for spatial data with different data distribution characteristics, wherein the data distribution characteristic of the spatial data is uniform distribution, or the data distribution characteristic of the spatial data is uneven distribution and the spatial data is in a clustered state. After acquiring the spatial data, the acquired spatial data is gridded and geographically partitioned according to a preset spatial indexing method that matches the data distribution characteristic of the spatial data, a spatial index file containing multiple feature object files is generated, and the multiple feature files in the spatial index file are stored in a distributed file system HDFS. When spatial data calculation is required, each computing node in Spark is called in parallel to enable each computing node to acquire multiple target feature object files with the same feature object file name from each storage node of the distributed file system HDFS, and the spatial data contained in the multiple target feature object files is used for data processing. The technical solution provided by the present invention has already gridded the spatial data before calculating it, so when calculating the spatial data, there is no need to geographically partition the spatial data, which reduces the time consumption of the calculation process. Moreover, when calculating the spatial data, by calling each computing node in Spark to perform parallel calculation, the time consumption of the calculation process is further reduced, thereby greatly improving the processing efficiency of the spatial data. That is to say, the present invention can realize the rapid distributed storage, management and analysis calculation of billions of spatial data.

[0067] Based on the distributed processing method of spatial big data disclosed in the embodiment of the present invention, the embodiment of the present invention also discloses a distributed processing device of spatial big data, such as Figure 4 As shown, the distributed processing device of spatial big data includes:

[0068] An acquisition unit 41 is used to acquire spatial data, where the spatial data includes vector data;

[0069] A first geographic partitioning unit 42 is configured to perform gridded geographic partitioning on the spatial data according to a preset spatial indexing method that matches the data distribution characteristics of the spatial data, thereby obtaining a spatial index file, wherein the spatial index file includes at least a plurality of feature object files, and the data distribution characteristics of the spatial data are uniform distribution, or the data distribution characteristics of the spatial data are non-uniform distribution and the spatial data is in a clustered state;

[0070] The storage unit 43 is used to store the multiple feature object files in the spatial index file in the distributed file system HDFS.

[0071] The specific principles and execution processes of each unit in the distributed processing device for spatial big data disclosed in the above embodiment of the present invention are the same as those of the distributed processing method for spatial big data disclosed in the above embodiment of the present invention. Please refer to the corresponding parts of the distributed processing method for spatial big data disclosed in the above embodiment of the present invention, and no further details will be given here.

[0072] The present invention provides a distributed processing device for spatial big data. A corresponding spatial indexing method is set for spatial data with different data distribution characteristics, wherein the data distribution characteristic of the spatial data is uniform distribution, or the data distribution characteristic of the spatial data is uneven distribution and the spatial data is in a clustered state. After acquiring the spatial data, the acquired spatial data is gridded and geographically partitioned according to a preset spatial indexing method that matches the data distribution characteristic of the spatial data, a spatial index file containing multiple feature object files is generated, and the multiple feature files in the spatial index file are stored in a distributed file system HDFS. When spatial data calculation is required, each computing node in Spark is called in parallel to enable each computing node to acquire multiple target feature object files with the same feature object file name from each storage node of the distributed file system HDFS, and the spatial data contained in the multiple target feature object files is used for data processing. The technical solution provided by the present invention has already gridded the spatial data before calculating it, so when calculating the spatial data, there is no need to geographically partition the spatial data, which reduces the time consumption of the calculation process. Moreover, when calculating the spatial data, by calling each computing node in Spark to perform parallel calculation, the time consumption of the calculation process is further reduced, thereby greatly improving the processing efficiency of the spatial data. That is to say, the present invention can realize the rapid distributed storage, management and analysis calculation of billions of spatial data.

[0073] Furthermore, each feature object file includes at least a feature object file name and spatial data corresponding to a geographical partition after gridding. The data processing device provided by the embodiment of the present invention further includes:

[0074] The execution unit is used to call the computing nodes in Spark in parallel to execute the corresponding steps. The corresponding steps executed by each computing node are:

[0075] A plurality of target feature object files having the same feature object file name are obtained from each storage node of the distributed file system HDFS, and data processing is performed using spatial data contained in the plurality of target feature object files.

[0076] Optionally, if the data distribution characteristic of the spatial data is uniform distribution, the first geographic partition unit includes:

[0077] The first determining unit is configured to determine that the spatial indexing method that matches the spatial data is a uniform grid spatial indexing method if the data distribution characteristic of the spatial data is uniform distribution;

[0078] The second geographic partitioning unit is used to perform gridded geographic partitioning on the spatial data based on a uniform grid spatial index method and generate a spatial index file.

[0079] Optionally, the second geographical division unit includes:

[0080] The third geographic partitioning unit is used to perform gridded geographic partitioning on the spatial data based on a uniform grid spatial index method to obtain a DSF feature dataset. The DSF feature dataset includes multiple geographic partitions and spatial data corresponding to each geographic partition.

[0081] The first generating unit is configured to generate, for each geographical partition, a feature object file corresponding to the geographical partition using the geographical partition and corresponding spatial data;

[0082] The second generating unit is configured to generate a spatial index file based on all feature object files.

[0083] Optionally, if the data distribution characteristics of the spatial data are uneven and the spatial data is in a clustered state, the first geographic partitioning unit includes:

[0084] The second determining unit is configured to determine that the spatial indexing method that matches the spatial data is a quadtree grid spatial indexing method if the data distribution characteristic of the spatial data is uneven distribution and the spatial data is in a clustered state;

[0085] The fourth geographic partitioning unit is used to perform gridded geographic partitioning on the spatial data based on a quadtree grid spatial indexing method and generate a spatial index file.

[0086] Optionally, the fourth geographical division unit includes:

[0087] A fifth geographic partitioning unit is configured to perform gridded geographic partitioning on the spatial data based on a quadtree grid spatial index method to obtain a DSF feature dataset, multiple geographic partitions of the DSF feature dataset, and spatial data corresponding to each geographic partition;

[0088] The third generating unit is configured to generate, for each geographical partition, a feature object file corresponding to the geographical partition using the geographical partition and the corresponding spatial data;

[0089] The fourth generating unit is configured to generate a spatial index file based on all the feature object files.

[0090] The present application embodiment provides an electronic device, such as Figure 5 As shown, the electronic device includes a processor 501 and a memory 502, the memory 502 is used to store program codes and data for data processing, and the processor 501 is used to call the program instructions in the memory to execute the steps shown in the distributed processing method of spatial big data disclosed in the above embodiment.

[0091] An embodiment of the present application provides a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the distributed processing method of spatial big data shown in the above embodiment.

[0092] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without making any creative efforts.

[0093] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0094] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

[0095] The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A distributed processing method for spatial big data, characterized in that: The method comprises: Acquiring spatial data, wherein the spatial data includes vector data; Performing gridded geographical partitioning on the spatial data according to a preset spatial indexing method that matches the data distribution characteristics of the spatial data to generate a spatial index file, wherein the spatial index file includes at least a plurality of feature object files, and the data distribution characteristics of the spatial data are uniform distribution, or the data distribution characteristics of the spatial data are uneven distribution and the spatial data is in a clustered state; Storing the plurality of feature object files in the spatial index file in a distributed file system HDFS; The step of performing gridded geographical partitioning on the spatial data and generating a spatial index file according to a preset spatial indexing method that matches the data distribution characteristics of the spatial data includes: If the data distribution characteristic of the spatial data is uniform distribution, determining that the spatial indexing method matching the spatial data is a uniform grid spatial indexing method; Based on the uniform grid spatial index method, the spatial data is subjected to grid geographical partitioning to generate a spatial index file; If the data distribution characteristic of the spatial data is uneven distribution and the spatial data is in a clustered state, determining that the spatial indexing method that matches the spatial data is a quadtree grid spatial indexing method; Based on the quadtree grid spatial indexing method, the spatial data is subjected to gridded geographical partitioning to generate a spatial index file.

2. The method according to claim 1, characterized in that Each of the feature object files includes at least a feature object file name and spatial data corresponding to a geographical partition after gridding the geographical partitions. After storing the plurality of feature object files in a distributed file system HDFS, the method further includes: The computing nodes in Spark are called in parallel to execute the corresponding steps. The corresponding steps executed by each computing node are: A plurality of target feature object files having the same feature object file name are obtained from each storage node of the distributed file system HDFS, and data processing is performed using the spatial data contained in the plurality of target feature object files.

3. The method according to claim 1, characterized in that The step of performing gridded geographical partitioning on the spatial data based on the uniform grid spatial indexing method to generate a spatial index file includes: Based on the uniform grid spatial index method, the spatial data is gridded and geographically partitioned to obtain a DSF feature dataset, wherein the DSF feature dataset includes multiple geographical partitions and spatial data corresponding to each geographical partition; For each of the geographical partitions, generating a feature object file corresponding to the geographical partition using the spatial data corresponding to the geographical partition; A spatial index file is generated based on all the feature object files.

4. The method according to claim 1, wherein The step of performing gridded geographical partitioning on the spatial data based on the quadtree grid spatial indexing method to generate a spatial index file includes: Based on the quadtree grid spatial indexing method, the spatial data is gridded and geographically partitioned to obtain a DSF feature dataset, a plurality of geographical partitions of the DSF feature dataset and spatial data corresponding to each geographical partition; For each of the geographical partitions, generating a feature object file corresponding to the geographical partition using the geographical partition and the corresponding spatial data; A spatial index file is generated based on all the feature object files.

5. A distributed processing device for spatial big data, characterized in that: The device comprises: an acquisition unit, configured to acquire spatial data, wherein the spatial data includes vector data; a first geographic partitioning unit, configured to perform gridded geographic partitioning on the spatial data according to a preset spatial indexing method that matches the data distribution characteristics of the spatial data, to obtain a spatial index file, wherein the spatial index file includes at least a plurality of feature object files, and the data distribution characteristics of the spatial data are uniform distribution, or the data distribution characteristics of the spatial data are non-uniform distribution and the spatial data is in a clustered state; A storage unit, configured to store the plurality of feature object files in the spatial index file in a distributed file system HDFS; The first geographical division unit is further configured to: If the data distribution characteristic of the spatial data is uniform distribution, determining that the spatial indexing method matching the spatial data is a uniform grid spatial indexing method; Based on the uniform grid spatial index method, the spatial data is subjected to grid geographical partitioning to generate a spatial index file; If the data distribution characteristic of the spatial data is uneven distribution and the spatial data is in a clustered state, determining that the spatial indexing method that matches the spatial data is a quadtree grid spatial indexing method; Based on the quadtree grid spatial indexing method, the spatial data is subjected to gridded geographical partitioning to generate a spatial index file.

6. The device according to claim 5, characterized in that Each of the feature object files at least includes a feature object file name and spatial data corresponding to a geographical partition after gridding, and the device further includes: The execution unit is used to call the computing nodes in Spark in parallel to execute the corresponding steps. The corresponding steps executed by each computing node are: A plurality of target feature object files having the same feature object file name are obtained from each storage node of the distributed file system HDFS, and data processing is performed using the spatial data contained in the plurality of target feature object files.

7. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory is used to store program code and data for distributed processing of spatial big data, and the processor is used to call the program instructions in the memory to execute a distributed processing method for spatial big data as described in any one of claims 1-4.

8. A storage medium, characterized in that: The storage medium includes a storage program, wherein when the program is running, the device where the storage medium is located is controlled to execute a distributed processing method for spatial big data according to any one of claims 1 to 4.

Images