Spark-based space incremental data processing method

Abstract

The invention discloses a spark-based spatial incremental data processing method, and belongs to the field of spatial data processing. Comprising the following steps: establishing an equalization space intelligent partitioning algorithm based on a spark parallel analysis principle; constructing a distributed storage format for the spatial data; during data reading, filtering the original partition data file by using the spatial index file; a spatial analysis mode is constructed based on an intelligent partitioning algorithm and a distributed storage format, and is used for executing the following steps: reading newly-added spatial data, and obtaining the data volume and the spatial range of the newly-added spatial data; carrying out space partitioning operation on the newly-added space data by utilizing an equalized space intelligent partitioning algorithm, and reducing each partition according to a data range; establishing a spatial secondary index in each partition; and writing the processed partition data into a database in parallel. The working efficiency of space incremental data processing can be improved, the labor cost is reduced, it is ensured that the processing process adopts a unified standard, human errors are reduced, and the accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of spatial data processing, and more specifically, to a spatial incremental data processing method based on Spark. Background Art

[0002] In data change surveys, the extraction of incremental spatial data is traditionally carried out using the C / S model. The problems of this model are the long spatial incremental data processing process, limited processing data volume, and low processing efficiency. In addition, it requires reliance on a variety of GIS desktop software, which increases the labor and economic costs of processing. Summary of the invention

[0003] The purpose of this invention is to propose a spatial incremental data processing method based on Spark, which improves the work efficiency of spatial incremental data processing by using big data technology, reduces labor costs, and ensures that the processing process adopts a unified standard, reduces human errors, and improves accuracy.

[0004] To achieve the above objectives, in a first aspect, the present invention proposes a method for processing spatial incremental data based on Spark, comprising: Expand support for spatial data in Spark; Based on the principle of Spark parallel analysis, a balanced spatial intelligent partitioning algorithm is established to partition spatial data based on the data volume and spatial range characteristics of spatial data; Construct a distributed storage format for spatial data, use a spatial storage index file and an original partition data file set to store the partitioned spatial data in a distributed file system; when reading data, use the spatial index file to filter the original partition data file to reduce the reading of data blocks; A spatial analysis mode is constructed based on the balanced spatial intelligent partitioning algorithm and the distributed storage format, and the spatial analysis mode is used to execute the following spatial incremental data processing flow: Read the newly added spatial data and obtain the data volume and spatial range of the newly added spatial data; Using the balanced space intelligent partitioning algorithm to perform space partitioning operations on the newly added space data, and reducing each partition according to the data range; Create a spatial secondary index in each partition; The processed partition data is written into the database in parallel to form the final spatial incremental data results.

[0005] Optionally, the balanced space intelligent partitioning algorithm includes: Perform spatial point sampling on the read spatial data and construct an input scale histogram to obtain the data volume and spatial range characteristics of the spatial data; Plan partitions based on the volume and spatial range characteristics of spatial data, and construct sampling partitions; Generate a sampling partitioner and perform partition operations on the real data; Each partition is reduced according to the data range, a data partitioner is generated and the partitioned data is obtained.

[0006] Optionally, after reducing each partition according to the data range, the following is further included: A data partitioner is generated and the partitioned first data is obtained.

[0007] Optionally, while performing a spatial partitioning operation on the newly added spatial data using the balanced spatial intelligent partitioning algorithm and reducing each partition according to the data range, it also includes: The basic data in the distributed file storage system is filtered according to the data range of the newly added spatial data and the spatial index file, and the filtered basic data is partitioned using the data partitioner to screen out the data that meets the conditions, thereby obtaining the partitioned second data.

[0008] Optionally, after creating a spatial secondary index in each partition, the following steps are also included: The partitioned first data and the second data are fused according to the partitions to obtain fused spatial data.

[0009] Optionally, after obtaining the fused spatial data, the method further includes: The fused data is subjected to spatial assertion judgment. If the assertion judgment conditions are met, the fused spatial data is recorded and the intersection is taken to extract the incremental data information.

[0010] Optionally, before extracting the incremental data information, the following is also included: Perform attribute processing on incremental data.

[0011] Optionally, the reading of the newly added spatial data includes: Define a reading range for spatial data, the reading range includes states, cities, districts and counties; The spatial data is read according to the reading range.

[0012] In a second aspect, the present invention provides an electronic device, the electronic device comprising: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Spark-based spatial incremental data processing method described in the first aspect.

[0013] In a third aspect, the present invention proposes a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the Spark-based spatial incremental data processing method described in the first aspect.

[0014] The beneficial effects of the present invention are: The invention discloses a method for processing spatial incremental data based on spark, and provides a balanced spatial intelligent partitioning technology based on the principle of spark parallel analysis, and partitions the data according to the data volume and spatial range characteristics of spatial data, and improves the parallelism and computing performance of subsequent calculations; at the same time, the invention designs a set of distributed storage formats for spatial data, and can use spatial index files to filter the original spatial data files during reading, and reduce the reading of data blocks in certain scenarios, and improve computing performance; further, the method of the invention also uses intelligent partitioning technology to cut large partitions after partitioning the incremental data, and further reduces the amount of data reading, and finally establishes a spatial secondary index in each partition, and accelerates the spatial analysis computing capability in each partition. The invention improves the traditional processing flow, uses big data technology to improve work efficiency, reduce labor costs, and ensure that the processing process adopts a unified standard, reduces human errors, and improves accuracy.

[0015] The system of the present invention has other characteristics and advantages, which will be apparent from the drawings incorporated herein and the following detailed description, or will be described in detail in the drawings incorporated herein and the following detailed description, which together serve to explain the specific principles of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, in which like reference numerals generally represent like components.

[0017] Figure 1 The present invention is a flowchart of distributed storage in a method for processing spatial incremental data based on Spark according to an embodiment of the present invention.

[0018] Figure 2 The present invention is a flowchart of spatial incremental data processing in a method for spatial incremental data processing based on Spark according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The present invention will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0020] Example 1

[0021] This embodiment provides a method for processing spatial incremental data based on Spark, including: S1: Expand support for spatial data in Spark; Specifically, the support for spatial data is expanded in Spark through functions such as Spark plug-ins, and a series of basic spatial analysis capabilities are provided at the same time.

[0022] S2: Based on the principle of Spark parallel analysis, a balanced spatial intelligent partitioning algorithm is established to partition spatial data based on the data volume and spatial range characteristics of spatial data; Specifically, a balanced spatial intelligent partitioning algorithm can be provided based on the spark parallel analysis principle to partition the data according to the data volume and spatial range characteristics of the spatial data, so as to improve the parallelism and computing performance of subsequent calculations.

[0023] In this embodiment, the balanced space intelligent partitioning algorithm includes: Perform spatial point sampling on the read spatial data and construct an input scale histogram to obtain the data volume and spatial range characteristics of the spatial data; Plan partitions based on the volume and spatial range characteristics of spatial data, and construct sampling partitions; Generate a sampling partitioner and perform partition operations on the real data; Each partition is reduced according to the data range, a data partitioner is generated and the partitioned data is obtained.

[0024] S3: construct a distributed storage format for spatial data, and use a spatial storage index file and an original partition data file set to store the partitioned spatial data in a distributed file system; when reading data, use the spatial index file to filter the original partition data file to reduce the reading of data blocks; Specifically, this method designs a distributed storage format for spatial data, which is mainly divided into spatial storage index files and original spatial data file sets. The spatial index files can be used to filter the original spatial data files during reading, which can reduce the reading of data blocks in some scenarios and improve computing performance.

[0025] In this embodiment, the spatial data distributed storage model implemented based on the constructed balanced spatial intelligent partitioning algorithm and distributed storage format is as follows: Figure 1 As shown, the specific process is as follows: First, input spatial data; Then, a balanced partitioning operation is performed: spatial point sampling is performed on the input spatial data, and an input scale histogram is constructed; Plan partitions by weight and construct sampling partitions; Generate a sampling partitioner and perform partition operations on the real data; Reduce each partition according to the data range, generate a data partitioner and obtain the partitioned data; After that, build the space storage index file and the original partition data file set; Finally, the partitioned spatial data is stored in a distributed file system.

[0026] S4: Construct a spatial analysis model based on the balanced spatial intelligent partitioning algorithm and the distributed storage format, such as Figure 2 As shown, the spatial analysis mode is used to execute the following spatial incremental data processing flow: Read the newly added spatial data and obtain the data volume and spatial range of the newly added spatial data through data evaluation; The balanced space intelligent partitioning algorithm is used to perform space partitioning operations on the newly added space data, and each partition is reduced according to the data range, a data partitioner is generated, and the first data after partitioning is obtained; at the same time, the basic data in the distributed file storage system is filtered according to the data range of the newly added space data and the space index file, and the data partitioner is used to partition the filtered basic data and filter out the data that meets the conditions, so as to obtain the second data after partitioning; Create a spatial secondary index in each partition; The partitioned first data and the second data are fused according to the partitions to obtain fused spatial data; The fused data is subjected to spatial assertion judgment. If the assertion judgment conditions are met, the fused spatial data is recorded and the intersection is taken to extract the incremental data information.

[0027] Perform attribute processing on incremental data; Finally, the processed partition data is written into the database in parallel to form the final spatial incremental data results.

[0028] Specifically, this method uses balanced spatial intelligent partitioning and distributed storage format to design a set of spatial analysis modes. First, the spatial data is read, and the user can customize the reading range, such as by state, city or district, and then spatial partitioning is performed, and large partitions are trimmed using intelligent partitioning technology to further reduce the amount of data read. Finally, spatial secondary indexes are established in each partition to accelerate the spatial analysis computing capabilities within each partition.

[0029] The method of the present invention adopts the high-performance distributed computing framework spark, expands the spatial type data, and provides a spatial incremental data processing method for spatial data cleaning and conversion, spatial data incremental extraction, and spatial data incremental library generation. Compared with the existing technology, the method uses big data technology to improve work efficiency, reduce labor costs, and ensure that the processing process adopts unified standards, reduces human errors, and improves accuracy.

[0030] Example 2

[0031] This embodiment provides an electronic device, the electronic device comprising: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Spark-based spatial incremental data processing method described in the above embodiment.

[0032] The electronic device according to an embodiment of the present disclosure includes a memory and a processor, the memory being used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0033] The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is used to execute the computer-readable instructions stored in the memory.

[0034] Those skilled in the art should be able to understand that in order to solve the technical problem of how to obtain a good user experience, the present embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the protection scope of the present disclosure.

[0035] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0036] Example 3

[0037] This embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the Spark-based spatial incremental data processing method described in the above embodiment.

[0038] According to the computer-readable storage medium of the embodiment of the present disclosure, non-transitory computer-readable instructions are stored thereon. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the above-mentioned methods of each embodiment of the present disclosure are executed.

[0039] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (e.g., memory card) and media with built-in ROM (e.g., ROM box).

[0040] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A method for processing spatial incremental data based on Spark, characterized in that: include: Expand support for spatial data in Spark; Based on the principle of Spark parallel analysis, a balanced spatial intelligent partitioning algorithm is established to partition spatial data based on the data volume and spatial range characteristics of spatial data; Construct a distributed storage format for spatial data, use a spatial storage index file and an original partition data file set to store the partitioned spatial data in a distributed file system; when reading data, use the spatial index file to filter the original partition data file to reduce the reading of data blocks; A spatial analysis mode is constructed based on the balanced spatial intelligent partitioning algorithm and the distributed storage format, and the spatial analysis mode is used to execute the following spatial incremental data processing flow: Read the newly added spatial data and obtain the data volume and spatial range of the newly added spatial data; Using the balanced space intelligent partitioning algorithm to perform space partitioning operations on the newly added space data, and reducing each partition according to the data range; Create a spatial secondary index in each partition; The processed partition data is written into the database in parallel to form the final spatial incremental data results.

2. The method for processing spatial incremental data based on Spark according to claim 1, characterized in that: The balanced space intelligent partitioning algorithm includes: Perform spatial point sampling on the read spatial data and construct an input scale histogram to obtain the data volume and spatial range characteristics of the spatial data; Plan partitions based on the volume and spatial range characteristics of spatial data, and construct sampling partitions; Generate a sampling partitioner and perform partition operations on the real data; Each partition is reduced according to the data range, a data partitioner is generated and the partitioned data is obtained.

3. The method for processing spatial incremental data based on Spark according to claim 1, characterized in that: After reducing each partition according to the data range, it also includes: A data partitioner is generated and the partitioned first data is obtained.

4. The method for processing spatial incremental data based on Spark according to claim 3 is characterized in that: While using the balanced space intelligent partitioning algorithm to perform space partitioning operations on the newly added space data and reducing each partition according to the data range, it also includes: The basic data in the distributed file storage system is filtered according to the data range of the newly added spatial data and the spatial index file, and the filtered basic data is partitioned using the data partitioner to screen out the data that meets the conditions, thereby obtaining the partitioned second data.

5. The method for processing spatial incremental data based on Spark according to claim 4 is characterized in that: After creating a spatial secondary index in each partition, it also includes: The partitioned first data and the second data are fused according to the partitions to obtain fused spatial data.

6. The method for processing spatial incremental data based on Spark according to claim 5, characterized in that: After obtaining the fused spatial data, it also includes: The fused data is subjected to spatial assertion judgment. If the assertion judgment conditions are met, the fused spatial data is recorded and the intersection is taken to extract the incremental data information.

7. The method for processing spatial incremental data based on Spark according to claim 6, characterized in that: Before extracting incremental data information, it also includes: Perform attribute processing on incremental data.

8. The method for processing spatial incremental data based on Spark according to claim 1, characterized in that: The reading of the newly added spatial data includes: Define a reading range for spatial data, the reading range includes states, cities, districts and counties; The spatial data is read according to the reading range.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Spark-based spatial incremental data processing method described in any one of claims 1-8.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the Spark-based spatial incremental data processing method described in any one of claims 1-8.

Images