Geographic big data processing method based on Spark

Abstract

The invention discloses a Spark-based geographic big data processing method, and relates to the technical field of data processing, and the method comprises the steps: obtaining a preset configuration file, and carrying out the configuration analysis of the configuration file, and obtaining an input parameter; obtaining geographical contour information according to the input parameters, analyzing the geographical contour information into a polygon object, and generating polygon object raster data according to the polygon object in a preset raster system; determining a data structure according to the data volume of the polygon object raster data, and converting the polygon object raster data into polygon data of the data structure; determining a storage mode according to the data volume of the polygon data and a preset memory resource of the computing node, and storing the polygon data according to the storage mode; and operating a data processing command defined in the input parameters to process the polygon data to obtain output data, and outputting the output data. The geographic big data calculation efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a Spark-based geographic big data processing method. Background Art

[0002] With the widespread application of geographic information systems (GIS) and the development of Internet of Things technology, the speed and amount of geographic big data generation have increased dramatically. Traditional data processing methods face performance bottlenecks when processing large-scale, high-dimensional geographic data, and it is difficult to meet the needs of real-time and high efficiency. In traditional technologies, the processing of geographic data such as point-to-region matching, region-to-region matching, and point-to-line matching cannot be efficiently matched. The Cartesian product traversal method is used to determine whether geographic elements intersect or are within a certain distance range, which results in large computational complexity and poor performance. There is no unified device for quickly realizing geographic big data calculation. In the face of different geographic big data processing needs, different programs need to be developed, with low development efficiency, uneven development quality, and high error probability. Faced with the characteristics of geographic data being often scattered, messy, and non-uniform standards, data processing has brought great challenges. Summary of the invention

[0003] The main purpose of this application is to provide a Spark-based geographic big data processing method, which is a Spark-based geographic big data processing method, device, equipment, medium and product, aiming to solve the technical problem of low efficiency of geographic big data calculation.

[0004] To achieve the above objectives, this application proposes a Spark-based geographic big data processing method, including:

[0005] Get the preset configuration file, parse the configuration file, and get the input parameters;

[0006] Acquire geographic contour information according to input parameters, parse the geographic contour information into polygonal objects, and generate polygonal object raster data according to the polygonal objects in a preset raster system, wherein the raster system divides the polygonal objects into a grid structure, and one grid structure corresponds to one grid;

[0007] Determine a data structure according to the data volume of the polygon object raster data, and convert the polygon object raster data into polygon data of the data structure, wherein the data structure includes a key-value pair and a double-layer key-value pair;

[0008] Determine a storage method according to the amount of polygon data and the memory resources of a preset computing node, and store the polygon data according to the storage method;

[0009] Run the data processing command defined in the input parameters to process the polygon data, obtain output data, and output the output data.

[0010] In one embodiment, the steps of acquiring geographic contour information according to input parameters, parsing the geographic contour information into polygon objects, and generating polygon object raster data according to the polygon objects in a preset grid system include:

[0011] Read geographic contour information according to the data storage address in the input parameter, and parse the geographic contour information into polygon objects;

[0012] Determine the circumscribed rectangle of each polygonal object according to the polygonal object, wherein the circumscribed rectangle is the smallest rectangle that contains the entire polygonal object;

[0013] Through the preset grid system, each grid in the circumscribed rectangle is intersected with the polygon object for topological calculation;

[0014] The polygon object grid data is determined for the intersecting grids according to the intersection topology calculation result, wherein the polygon object grid data includes a data set of the polygon object and a data set of the grids intersecting with the polygon object.

[0015] In one embodiment, the step of determining the data structure according to the data volume of the polygon object grid data includes:

[0016] Determine the data volume of the polygon object raster data;

[0017] If the data volume is less than a preset data volume threshold, the data structure is determined to be a key-value pair, wherein the key of the key-value pair includes a grid index value, and the value of the key-value pair is a data set of all polygon objects covered by the grid corresponding to the grid index value, wherein the grid index value is used to represent the position of the grid, including the grid side length, the grid longitude index value, and the grid latitude index value;

[0018] If the data volume is greater than or equal to the data volume threshold, the data structure is determined to be a double-layer key-value pair, wherein the first layer key of the double-layer key-value pair is the grid longitude index value, the second layer key of the double-layer key-value pair is the grid latitude index value, and the value of the double-layer key-value pair is the sub-polygon object of the grid corresponding to the grid longitude index value and the grid latitude index value, and the sub-polygon object is the contour data of the part where each raster intersects with the polygon object.

[0019] In one embodiment, in a preset grid system, before the step of generating polygon object grid data according to the polygon object, the method further includes:

[0020] Determine the preset standard meridian length, determine the latitude difference per degree according to the standard meridian length, and determine the latitude difference per meter according to the latitude difference per degree, wherein the latitude difference per degree is the actual distance corresponding to each degree of latitude, and the latitude difference per meter is the latitude change corresponding to each meter;

[0021] Determine the preset center point latitude value, determine the longitude difference per degree according to the center point latitude value and the latitude difference per degree, and determine the longitude difference per meter according to the longitude difference per degree, wherein the longitude difference per degree is the actual distance corresponding to each degree of longitude, and the longitude difference per meter is the longitude change corresponding to each meter;

[0022] For each point data in the polygon object raster data, determine the longitude and latitude of the point data;

[0023] Determine the normalized longitude code of the point data according to the longitude per meter difference and the longitude, wherein the normalized longitude code is used to represent the position of the longitude in the grid system;

[0024] Determine the latitude normalization code of the point data according to the latitude per meter difference and the latitude, wherein the latitude normalization code is used to represent the position of the latitude in the grid system;

[0025] The normalized longitude code is used as the raster longitude index value, the normalized latitude code is used as the latitude longitude index value, and the raster index value is generated according to the raster longitude index value and the latitude longitude index value.

[0026] In one embodiment, the step of determining the storage method according to the amount of polygon data and the memory resources of the preset computing node includes:

[0027] Determine the amount of polygon data and determine whether the memory resources of the computing node are less than a preset storage threshold;

[0028] If the amount of polygon data is less than the storage threshold and the memory resources of the computing node are sufficient, the storage method is determined to store the polygon data in a preset total server memory, wherein the polygon data will be broadcast and saved to each computing node before calculation;

[0029] If the polygon data is greater than or equal to the storage threshold or the memory resources of the computing node are insufficient, the storage method is determined to convert the polygon data into a resilient distributed dataset for storage, wherein the resilient distributed dataset has multiple storage points.

[0030] In one embodiment, the input parameters include user-defined function information, and the step of running the data processing command defined in the input parameters to process the polygon data and obtain the output data includes:

[0031] Register the custom function according to the custom function information and bind the polygon data to the custom function;

[0032] Run the data processing command defined in the input parameters to process the polygon data. The steps to obtain the output data include:

[0033] By executing the data processing command and calling the custom function, the polygon data is processed to obtain the output data.

[0034] In addition, to achieve the above purpose, the present application also proposes a Spark-based geographic big data processing device, which includes:

[0035] Get parameter module, get preset configuration file, and parse the configuration file to get input parameters;

[0036] A rasterization processing module obtains geographic contour information according to input parameters, parses the geographic contour information into polygonal objects, and generates polygonal object raster data according to the polygonal objects in a preset raster system, wherein the raster system divides the polygonal objects into a grid structure, and one grid structure corresponds to one grid;

[0037] A data structure definition module determines a data structure according to the data volume of the polygon object raster data, and converts the polygon object raster data into polygon data of the data structure, wherein the data structure includes a key-value pair and a double-layer key-value pair;

[0038] A storage mode determination module determines a storage mode according to the amount of polygon data and the memory resources of a preset computing node, and stores the polygon data according to the storage mode;

[0039] The data processing module runs the data processing command defined in the input parameters to process the polygon data, obtains output data, and outputs the output data. In addition, to achieve the above purpose, the present application also proposes a Spark-based geographic big data processing device, the device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program is configured to implement the steps of the Spark-based geographic big data processing method as described above.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a medium, which is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the Spark-based geographic big data processing method as described above are implemented.

[0041] In addition, to achieve the above-mentioned objectives, the present application also provides a product, which is a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps of the Spark-based geographic big data processing method as described above.

[0042] One or more technical solutions proposed in this application have at least the following technical effects:

[0043] The present application obtains a preset configuration file, and performs configuration and analysis on the configuration file to obtain input parameters. This step can flexibly configure the data processing flow and improve the scalability and maintainability of the system; obtains geographic contour information according to the input parameters, parses the geographic contour information into polygon objects, and generates polygon object raster data according to the polygon objects in a preset raster system, wherein the raster system divides the polygon objects into a grid structure, one grid structure corresponds to one raster, and through rasterization processing, the geographic elements are converted into a form that is easier to calculate and search, laying the foundation for subsequent efficient processing; determines the data structure according to the data volume of the polygon object raster data, and converts the polygon object raster data into polygon data of the data structure, wherein the data structure includes key-value pairs and double-layer key-value pairs, and the design of this data structure can optimize data storage The storage and access efficiency can be improved to improve data processing performance; the storage method is determined according to the amount of polygon data and the memory resources of the preset computing nodes, and the polygon data is stored according to the storage method, which can reasonably utilize computing resources, avoid memory overflow or resource waste, and improve the stability and processing efficiency of the system; the data processing command defined in the input parameters is run to process the polygon data to obtain output data, and the output data is output. By running the custom data processing command, various processing requirements for polygon data can be quickly and flexibly realized, such as calculating the area, perimeter, screening specific areas, etc. At the same time, due to the use of targeted data structure and storage method design and encapsulation of custom functions, the computing efficiency is greatly improved and the amount of calculation is greatly reduced. Compared with the traditional Cartesian product cross traversal calculation method, it has higher efficiency and lower complexity. In addition, a general Spark-based geographic big data processing program is constructed, so that in the face of various Spark-based geographic big data processing requirements, only data processing commands need to be written and the appropriate geographic custom functions can be selected to quickly achieve them, which reduces development costs and greatly improves development efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 This is a flow chart of Example 1 of the geographic big data processing method based on Spark in this application;

[0047] Figure 2 This is a small area map schematic diagram of the Spark-based geographic big data processing method for this application;

[0048] Figure 3 This is a key-value pair diagram of the Spark-based geographic big data processing method of this application;

[0049] Figure 4 A schematic diagram of a double-layer key-value pair of the Spark-based geographic big data processing method of this application;

[0050] Figure 5 A large-area map schematic diagram of the Spark-based geographic big data processing method for this application;

[0051] Figure 6 A schematic diagram of a sub-polygon of the Spark-based geographic big data processing method of this application;

[0052] Figure 7 This is a schematic diagram of sub-polygon storage of the Spark-based geographic big data processing method of this application;

[0053] Figure 8 A schematic diagram of broadcast variables of the Spark-based geographic big data processing method of this application;

[0054] Fig. 9 This is a schematic diagram of the module structure of a geographic big data processing device based on Spark in an embodiment of the present application;

[0055] Fig.10 This is a schematic diagram of the device structure of the hardware operating environment involved in the Spark-based geographic big data processing method in the embodiment of the present application.

[0056] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0058] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0059] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a terminal system, etc. The following takes the system as an example to illustrate this embodiment and the following embodiments.

[0060] Based on this, this embodiment provides a Spark-based geographic big data processing method. Figure 1 , Figure 1 This is a flow chart of the Spark-based geographic big data processing method of this application. The Spark-based geographic big data processing method includes steps S10 to S40:

[0061] Step S10, obtaining a preset configuration file, and performing configuration analysis on the configuration file to obtain input parameters;

[0062] It should be noted that the configuration file is a file that contains various parameters and settings required for data processing tasks, such as the storage location of input data, data format, processing logic, etc. Configuration parsing is to read the configuration file and parse the parameters and settings in it for use in subsequent steps.

[0063] The system first reads the configuration file preset by the user, which contains various parameters and settings required for data processing tasks, such as the storage location of input data, data format, processing logic, etc. Then, the system parses the configuration file and extracts these parameters and settings as input parameters for subsequent steps.

[0064] For example, this embodiment introduces the yaml (a file format) file format as a configuration tool for the computing process, which greatly simplifies the complexity of the operation. Developers only need to specify the attribute parameters of the geographic dataset in the yaml file and write the corresponding business logic SQL (Structured Query Language) statement. The system can parse the output data configured by the developer based on the yaml file.

[0065] A complete data processing process can be abstracted as: data input (reader), calculation processing (processor), and result output (writer). Therefore, a task file should generally contain the above three modules, which are introduced in Table 1 as examples below:

[0066]

[0067]

[0068] Table 1

[0069] Step S20, obtaining geographic contour information according to the input parameters, parsing the geographic contour information into polygonal objects, and generating polygonal object raster data according to the polygonal objects in a preset grid system, wherein the grid system divides the polygonal objects into a grid structure, and one grid structure corresponds to one grid;

[0070] It should be noted that geographic contour information is data that describes the location and shape of geographic elements (such as points, lines, and surfaces), such as the vertex coordinate sequence of a polygon. A polygon object is a data structure that represents geographic contour information in a computer, and usually contains vertex coordinate information and polygon type information. A raster system is a system that divides geographic space into a series of grid-like structures. Each grid is called a raster, which is used to index and store geographic data. Polygon object raster data is raster index data generated in a raster system based on the shape and location of a polygon object, and is used to quickly find rasters related to polygons.

[0071] According to the information in the input parameters, the system reads the geographic contour information from the specified data storage location. Then, the system parses the geographic contour information into polygon objects, which contain the coordinate information of the vertices and the type information of the polygon. Then, in the preset grid system, the system generates polygon object raster data according to the shape and position of the polygon object. The essence of this step is to establish a spatial index for the polygon object, that is, to determine the position of the polygon object in the grid system and generate the corresponding grid index data.

[0072] Step S30, determining a data structure according to the data volume of the polygon object raster data, and converting the polygon object raster data into polygon data of the data structure, wherein the data structure includes key-value pairs and double-layer key-value pairs;

[0073] It should be noted that data structure is a way to organize, manage and store data in a computer, such as key-value pairs, double-layer key-value pairs, etc.

[0074] The system selects a suitable data structure for storage based on the amount of polygon object raster data. For small and medium-sized data sets, a key-value pair structure can be selected; for large data sets, a double-layer key-value pair structure can be selected to improve search efficiency. The system then converts the polygon object raster data into polygon data of the selected data structure for subsequent processing.

[0075] Step S40, determining a storage method according to the amount of polygon data and the preset memory resources of the computing node, and storing the polygon data according to the storage method;

[0076] It should be noted that the storage method refers to the storage form of data in computer memory or disk, such as broadcast variables, RDD / DataSet (Resilient Distributed Dataset), etc.

[0077] The system determines the data storage method based on the amount of polygon data and the memory resources of the preset computing nodes. If the memory resources are sufficient and the amount of data is not large, you can choose to store the data as a broadcast variable so that it can be quickly accessed during calculation; if the amount of data is large or the memory resources are limited, you can choose to store the data as a distributed data structure such as RDD / DataSet. Then, the system stores the polygon data in the corresponding memory or disk location based on the selected storage method.

[0078] Step S50, running the data processing command defined in the input parameters to process the polygon data, obtaining output data, and outputting the output data.

[0079] It should be noted that data processing commands are instructions or scripts that define data processing logic and steps, such as SQL statements.

[0080] The system processes the stored polygon data according to the data processing commands (such as SQL statements) defined in the input parameters. These commands define the specific logic and steps of data processing, such as screening, calculation, conversion, etc. After processing, the system obtains output data. These output data can be geographical data after screening, calculation or conversion, or statistical information or other forms of data of the processing results. Finally, the system saves the output data to a specified location for subsequent use or analysis.

[0081] This embodiment obtains a preset configuration file, and performs configuration analysis on the configuration file to obtain input parameters. This step can flexibly configure the data processing flow and improve the scalability and maintainability of the system; obtains geographic contour information according to the input parameters, analyzes the geographic contour information into polygon objects, and generates polygon object raster data according to the polygon objects in a preset grid system, wherein the grid system divides the polygon objects into a grid structure, one grid structure corresponds to one grid, and through rasterization processing, the geographic elements are converted into a form that is easier to calculate and search, laying the foundation for subsequent efficient processing; determines the data structure according to the data volume of the polygon object raster data, and converts the polygon object raster data into polygon data of the data structure, wherein the data structure includes key-value pairs and double-layer key-value pairs, and the design of this data structure can optimize data storage The storage and access efficiency can be improved to improve data processing performance; the storage method is determined according to the amount of polygon data and the memory resources of the preset computing nodes, and the polygon data is stored according to the storage method, which can reasonably utilize computing resources, avoid memory overflow or resource waste, and improve the stability and processing efficiency of the system; the data processing command defined in the input parameters is run to process the polygon data to obtain output data, and the output data is output. By running the custom data processing command, various processing requirements for polygon data can be quickly and flexibly realized, such as calculating area, perimeter, screening specific areas, etc. At the same time, due to the use of targeted data structure and storage method design and encapsulation of custom functions, the calculation efficiency is greatly improved and the calculation amount is greatly reduced. Compared with the traditional Cartesian product cross traversal calculation method, it has higher efficiency and lower complexity. In addition, the construction of a general geographic big data processing program enables the face of various geographic big data processing needs, only need to write data processing commands, select the appropriate geographic custom function to quickly achieve, which reduces the development cost and greatly improves the development efficiency and quality.

[0082] In a feasible implementation manner, the steps in step S50 and the steps before step S50 may include steps T10 to T20:

[0083] Step T10, registering the custom function according to the custom function information, and binding the polygon data to the custom function;

[0084] It should be noted that the user-defined function information refers to the function description information defined by the user for a specific data processing task, including the function name, parameter type, return value type, etc. Registering a user-defined function is to add the user-defined function information to the function library of the system or framework so that it can be recognized and called by the system. In this embodiment, binding refers to associating polygon data with the user-defined function so that the polygon data can be processed by the user-defined function when needed.

[0085] First, the user needs to provide custom function information, which is received by the system and registered in the function library. Then, the user specifies which polygon data to bind to this custom function. The binding operation is usually implemented by referencing the function name in the data processing command. Registering custom functions is a common way to expand the processing capabilities of the system. Through registration, the system can recognize and call user-defined functions to achieve custom processing of polygon data. The binding operation ensures that when the data processing command is executed, the correct function will be applied to the corresponding data.

[0086] Step T20, calling the custom function by executing the data processing command, processes the polygon data to obtain output data.

[0087] It should be noted that output data refers to polygon data processed by custom functions, and usually contains the results or information required by the user.

[0088] In the input parameters, the user defines the data processing commands, which specify what processing to do with the polygon data, and can be implemented by writing the corresponding business logic SQL (Structured Query Language) statements. When the system receives these commands, it will find the custom functions associated with these commands and call these functions to process the polygon data. The processing process may involve data conversion, filtering, calculation or analysis, etc. The data processing command is a bridge for the user to interact with the system, and it guides the system on how to process the polygon data. By calling the custom function, the system can execute the user-defined processing logic to generate output data that meets the user's needs. This process reflects the high flexibility and scalability of the system. After being processed by the custom function, the polygon data is converted into output data. These output data may contain various information required by the user, such as the area, perimeter, vertex coordinates, etc. of the polygon. The output data is the final result of the data processing process, which reflects the processing effect of the custom function on the polygon data. By checking and analyzing the output data, the user can evaluate the correctness and effectiveness of the custom function and make subsequent data processing or decisions based on it.

[0089] Exemplarily, in a Spark (a big data processing framework) environment, after completing the environment configuration, the user uses the spark-submit command to specify the resource configuration and task name (i.e., the configuration file directory name). This embodiment can submit the Spark task according to the configured data processing flow, obtain data according to the user's configuration, register custom functions, and complete the data processing task according to the user-configured data processing commands.

[0090] This embodiment realizes the customized processing of polygon data, and improves the flexibility and scalability of data processing. Users can define processing logic according to their own needs, and apply these logics to the actual data processing process through registration and binding operations.

[0091] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above introduction, and will not be repeated hereafter. On this basis, the steps of step S20 also include steps A10 to A40:

[0092] Step A10, reading geographic contour information according to the data storage address in the input parameter, and parsing the geographic contour information into polygon objects;

[0093] Read geographic contour information according to the data storage address in the input parameter. This information is usually stored in the form of files, such as WKT (Well-Known Text) format, GeoJSON (a format for encoding various geographic data structures), etc. Then, parse this information into polygon objects, that is, construct the vertex coordinate sequence and shape type of the polygon. This step is the key to data preprocessing, converting the original geographic data into computer-recognizable polygon objects, laying the foundation for subsequent spatial indexing and calculations.

[0094] Step A20, determining the circumscribed rectangle of each polygonal object according to the polygonal object, wherein the circumscribed rectangle is the smallest rectangle that contains the entire polygonal object;

[0095] It should be noted that the bounding rectangle is the smallest rectangle that contains the entire polygon object and is used to simplify spatial indexing and query operations.

[0096] For each polygon object, calculate its bounding rectangle. The bounding rectangle is determined by finding the minimum longitude and minimum latitude among all the vertices of the polygon as the lower left corner of the rectangle, and the maximum longitude and maximum latitude as the upper right corner of the rectangle. The bounding rectangle is an effective way to simplify spatial indexing. By wrapping the polygon object in a rectangular frame, the coordinate range of the rectangle can be conveniently used for spatial query and indexing, thereby improving data processing efficiency.

[0097] Step A30, performing intersection topology calculation on each grid in the circumscribed rectangle and the polygon object through a preset grid system;

[0098] It should be noted that a raster is a system that divides geographic space into a series of rectangular units of equal size. Each unit is called a raster and is used for spatial indexing and data processing. Intersection topology calculation is to determine whether two geographic elements (such as rasters and polygons) intersect in space and calculate the topological relationship of the intersecting parts (such as intersection points, intersection lines, and intersection surfaces).

[0099] For each polygon object's bounding rectangle, traverse each grid in it. Then, use the intersection topology calculation algorithm to determine whether each grid intersects with the polygon object. Intersection topology calculation is the core technology of spatial indexing and querying. By determining the intersection relationship between grids and polygons, it is possible to determine which grids contain part or all of the polygons, thus providing a basis for subsequent spatial analysis and data processing.

[0100] Step A40, determining polygonal object raster data for the intersecting grids according to the intersection topology calculation result, wherein the polygonal object raster data includes a data set of the polygonal object and a data set of the grids intersecting with the polygonal object.

[0101] All grids that intersect with polygon objects are extracted from the results of intersection topology calculation, and together with the polygon object data set, they form a polygon object grid data set. This set contains the index information of all grids that intersect with polygon objects in space. Polygon object grid data is the basis for subsequent spatial analysis and data processing. By associating these grids with polygon objects, spatial query, spatial analysis and other operations can be easily performed, such as calculating polygon area, perimeter, distance, etc.

[0102] This embodiment realizes the effective processing of geographic contour information and the construction of spatial index. First, the geographic contour information is read and parsed as a polygon object according to the input parameters, providing basic data for subsequent operations. Next, the bounding rectangle of the polygon object is determined, which simplifies the spatial index and query operations. Then, the intersection topology calculation is performed through the preset grid system, and the intersection relationship between the polygon and the grid is accurately determined, providing key information for subsequent spatial analysis and data processing. Finally, based on the results of the intersection topology calculation, the polygon object raster data is determined. This data set contains detailed information about the polygon and its intersection grid, which provides strong support for subsequent spatial query, analysis and calculation operations, and significantly improves data processing efficiency and accuracy.

[0103] In a feasible implementation manner, step S30 further includes steps B10 to B30:

[0104] Step B10, determining the data volume of the polygon object raster data;

[0105] It should be noted that the data volume refers to the number of grids or the total amount of data in the polygon object raster data, which is used to evaluate the scale and complexity of the data.

[0106] Step B20, if the data volume is less than a preset data volume threshold, determine that the data structure is a key-value pair, wherein the key of the key-value pair includes a grid index value, and the value of the key-value pair is a data set of all polygon objects covered by the grid corresponding to the grid index value, wherein the grid index value is used to represent the position of the grid, including the grid side length, the grid longitude index value, and the grid latitude index value;

[0107] It should be noted that the data volume threshold is a preset upper limit of the data volume, which is used to determine which data structure is used to store and process polygon object raster data. A key-value pair is a data structure consisting of a key (Key) and a value (Value), which is used to store and quickly retrieve data. In this embodiment, the key is a grid index value, and the value is a data set of polygon objects covered by the grid. The grid index value is a value used to uniquely identify the grid position, including the grid side length (usually fixed, not used for unique identification), the grid longitude index value, and the grid latitude index value.

[0108] Calculate the total number of grids or the total amount of data in the polygon object raster data. If the amount of data is less than the preset data amount threshold, select the key-value pair as the data structure. The key is the grid index value (including grid edge length, grid longitude index value, grid latitude index value). Sometimes, since the grid edge length is usually fixed, it may not be included in the grid index value. The value is the data set of all polygon objects covered by the grid. When the amount of data is small, the key-value pair structure can efficiently store and retrieve data, and is easy to implement and maintain.

[0109] For example, for a data set of relatively small polygonal objects, since a single grid (whose side length is set to 400 meters by default, but can be flexibly adjusted according to actual application scenarios) can usually cover multiple polygonal objects, a one-to-many mapping relationship between grids and polygons is formed. Figure 2 As shown, the red border is a polygon and the green border is a grid. To achieve efficient management of this mapping, we use a list structure based on key-value pairs. <index,List <polygon>>, where index (key) represents the index value of the grid, and the value is a List <polygon>, which stores a collection of all polygon objects covered by the grid. Figure 3 , Figure 3 This is a diagram of key-value pairs. The key 0_20 corresponds to polygon object A, the key 20_20 corresponds to polygon object A and polygon object B, and the key 40_20 corresponds to empty.

[0110] Step B30, if the data volume is greater than or equal to the data volume threshold, the data structure is determined to be a double-layer key-value pair, wherein the first layer key of the double-layer key-value pair is the grid longitude index value, the second layer key of the double-layer key-value pair is the grid latitude index value, and the value of the double-layer key-value pair is the sub-polygon object of the grid corresponding to the grid longitude index value and the grid latitude index value, and the sub-polygon object is the contour data of the part where each grid intersects with the polygon object.

[0111] It should be noted that the double-layer key-value pair is a complex data structure consisting of two layers of key-value pairs, which is used to store more complex data relationships. In this embodiment, the first layer key is the grid longitude index value, and the second layer key is the grid latitude index value, and the value is the sub-polygon object of the corresponding grid. The sub-polygon object refers to the contour data of the intersection of each grid and the polygon object, and is a subdivision representation of the polygon object on the grid system.

[0112] If the amount of data is greater than or equal to the data volume threshold, a double-layer key-value pair is selected as the data structure. The first-layer key is the grid longitude index value, and the second-layer key is the grid latitude index value. The value is the sub-polygon object of the corresponding grid. The sub-polygon object is the outline data of the intersection of each grid and the polygon object, which can accurately represent the distribution of polygons on the grid system. When the amount of data is large, the double-layer key-value pair structure can manage complex data relationships more efficiently and reduce the time complexity of data retrieval through hierarchical indexing.

[0113] For example, in view of the possible search efficiency problems caused by the direct indexing method of single-layer key-value pairs when fully matching two-dimensional information, we further propose an optimized double-layer key-value pair structure, which splits the key into longitude X and latitude Y, that is, <X,<Y,List <polygon>>>. This structure allows for preliminary screening at a coarser level by introducing an additional X level, followed by precise matching using Y. This design significantly reduces unnecessary search space, allowing the search process to be quickly exited when no match is found, thereby greatly improving overall search efficiency. Converting the key-value pairs in the example of step B20 into a double-layer key-value pair data structure yields Figure 4 Schematic diagram of the double-layer key-value pair, key (X) is 0, key (Y) is 20, the corresponding value is polygon object A, key (X) is 20, key (Y) is 20, the corresponding value is polygon object A and polygon object B, key (X) is 40, key (Y) is 20, the corresponding value is empty. However, for polygon datasets with large areas and long point sequences, such as administrative districts and large scenic areas, a single polygon object may contain tens of thousands of grids. If it is stored according to the key-value pair data structure mentioned above, the same polygon object will be stored under a large number of keys, resulting in a waste of memory resources; and because the polygon point sequence is long, the time spent on calculating the intersection relationship will also increase. Figure 5 As shown, this embodiment optimizes this type of scene. The rasterization and key-value pair storage methods remain unchanged, but the values ​​are no longer stored in polygon lists. Instead, the polygons are defined as sub-polygons, such as Figure 6 As shown in the figure, it is a schematic diagram of a sub-polygon. A polygon forms multiple sub-polygons 01, 02, 03, and 04 in the grid system. When storing data, the double-layer key-value pairs are kept unchanged, and the polygon object is converted into a sub-polygon object. The data structure is <X,<Y,List <polygonintersectionitem>>>。List <polygonintersectionitem>A data set of sub-polygon objects. Figure 7 , Figure 7 This is a schematic diagram of sub-polygon storage. The key (X) is 20, the key (Y) is 20 and the corresponding value is sub-polygon object 01. The key (X) is 40, the key (Y) is 20 and the corresponding value is sub-polygon object 02.

[0114] This embodiment can flexibly select the data structure according to the data volume of the polygon object raster data, thereby optimizing the data storage and processing performance, and providing efficient data support for subsequent spatial analysis and data processing. The key-value pair data structure is suitable for situations with small data volumes, and can quickly retrieve and obtain polygon object data covered by a specific grid. The double-layer key-value pair data structure is suitable for situations with large data volumes, and improves data retrieval efficiency through hierarchical indexing. At the same time, it can accurately represent the distribution of polygons on the grid system, providing convenience for subsequent spatial analysis and data processing.

[0115] In a feasible implementation manner, before step S20, steps C10 to C60 are also included:

[0116] Step C10, determining the preset standard meridian length, determining the latitude difference per degree according to the standard meridian length, and determining the latitude difference per meter according to the latitude difference per degree, wherein the latitude difference per degree is the actual distance corresponding to each degree of latitude, and the latitude difference per meter is the latitude change corresponding to each meter;

[0117] It should be noted that the standard meridian length is in geography, usually refers to the length of the meridian on the equator, which is a fixed value used to calculate the difference in longitude and latitude at other locations on the earth's surface. Two opposite meridians form a meridian circle. The length of a meridian circle = the length of the equator = 40,000 kilometers, so the length of a meridian is about 20,000 kilometers. Meridians, like latitudes, are auxiliary lines assumed by humans for the convenience of measurement. They are defined as semicircular arcs on the great circle connecting the North and South Poles on the earth's surface. Any two meridians are equal in length and intersect at the North and South Poles. Each meridian has its corresponding value, called longitude, indicating the east-west position. It is about 20037km. The difference per degree of latitude is the actual distance corresponding to each degree of latitude change on the earth's surface. This value changes with the change of latitude. The difference per meter of latitude is the angle difference corresponding to each meter of latitude change at a specific latitude, which is used to accurately calculate the latitude position.

[0118] First, determine a fixed standard meridian length (such as the length of the meridian at the equator). Then, based on the shape and size of the earth, calculate the actual distance corresponding to each degree of latitude change at a specific latitude (latitude difference per degree), and further calculate the angle difference corresponding to each meter of latitude change (latitude difference per meter).

[0119] Step C20, determining the preset center point latitude value, determining the longitude difference per degree according to the center point latitude value and the latitude difference per degree, and determining the longitude difference per meter according to the longitude difference per degree, wherein the longitude difference per degree is the actual distance corresponding to each degree of longitude, and the longitude difference per meter is the longitude change corresponding to each meter;

[0120] It should be noted that the center point latitude value refers to the latitude value of the center point of a certain geographic area or grid system, which is used to calculate the longitude difference of the area or system. The longitude difference per degree is the actual distance corresponding to one degree change in longitude at a specific latitude. This value changes with the change of latitude. The longitude difference per meter is the angle difference corresponding to one meter change in longitude at a specific latitude and longitude, which is used to accurately calculate the longitude position.

[0121] Select a center point latitude value, which can be the latitude of the center point of a geographic area or grid system. Then, based on the center point latitude value, calculate the actual distance corresponding to each degree of longitude change at that latitude (longitude per degree difference), and further calculate the angle difference corresponding to each meter of longitude change (longitude per meter difference).

[0122] Step C30, for each point data in the polygon object raster data, determine the longitude and latitude of the point data;

[0123] For each point data in the polygon object raster data, extract its longitude and latitude values.

[0124] Step C40, determining the normalized longitude code of the point data according to the longitude per meter difference and the longitude, wherein the normalized longitude code is used to characterize the position of the longitude in the grid system;

[0125] It should be noted that the normalized longitude code is to convert the longitude value into a code that uniquely identifies the location in the grid system, usually obtained by dividing the longitude value by a reference value (such as the longitude per meter difference) and rounding it.

[0126] Using the longitude per meter difference as the reference value, the longitude value of each point data is divided by this reference value and rounded to obtain the longitude normalization code. This code is used to uniquely identify the longitude position of the point data in the grid system.

[0127] Step C50, determining the latitude normalization code of the point data according to the latitude per meter difference and the latitude, wherein the latitude normalization code is used to represent the position of the latitude in the grid system;

[0128] It should be noted that the latitude normalization code is to convert the latitude value into a code that uniquely identifies the location in the grid system, usually obtained by dividing the latitude value by a reference value (such as the latitude difference per meter) and rounding it.

[0129] Using the latitude per meter difference as the reference value, the latitude value of each point data is divided by this reference value and rounded to get the latitude normalization code. This code is used to uniquely identify the latitude position of the point data in the raster system.

[0130] Step C60, using the normalized longitude code as the grid longitude index value, using the normalized latitude code as the latitude longitude index value, and generating the grid index value according to the grid longitude index value and the latitude longitude index value.

[0131] It should be noted that the grid index value is a value composed of the normalized longitude code and the normalized latitude code, and is used to uniquely identify a grid position in the grid system.

[0132] The normalized longitude and latitude codes of each point data are combined to form a unique grid index value, which is used to uniquely identify the grid location of the point data in the grid system.

[0133] For example, the lengths of the earth's meridians (standard meridian lengths) are all equal, and the standard meridian length Le is 20037000 meters, so the latitude difference L per degree is lat_diff =L e / 180=111000, we can get the latitude difference per meter α lat =1 / L lat_diff ≈0.000009°, which means the proportion of 1 meter in the ground distance corresponding to 1 degree of latitude change. The latitude value of the location of the design calculation point is Y mid , China's latitude spans from 4°N to 53°N, taking the middle value, the center point latitude value Y loc =28.5°, so the difference in longitude per degree can be obtained by formula (1):

[0134] L lng_diff =L lat_diff ×cosY loc Formula (I)

[0135] Among them, L lng_diff is the difference in longitude per degree, L lat_diff is the latitude difference per degree, cosY loc is the cosine function, where Y loc is the latitude value of a particular location. The cosine function is used to adjust the effect of the length of the longitude line changing with latitude. The difference in longitude per degree can be obtained from the difference in longitude per meter α lng =1 / L lng_diff =0.0000102512≈0.000010°, which means the proportion of 1 meter in the ground distance corresponding to a change of 1 degree in longitude.

[0136] Let the side length of the grid (square) be L grid , the longitude of the point data to be calculated is X, and the latitude is Y. The calculation formula for the normalized longitude code is shown in formula (2), and the calculation formula for the normalized latitude code is shown in formula (3):

[0137]

[0138] Among them, X round is the normalized code of longitude, which indicates the index of longitude in the grid system. In formula (2), firstly, the longitude X is divided by the longitude difference per meter α lng and grid side length L grid The product of , gets the relative position of longitude in the grid system. Then, use the floor function to get the normalized longitude code, that is, the grid longitude index. Similarly, formula (3) also gets the latitude normalized code Y in a similar way round The grid index value of the final output coordinate (X, Y) is L grid _X round _Y round .

[0139] This embodiment can accurately map each point data in the polygon object raster data into the raster system and generate a unique raster index value for it, which facilitates subsequent spatial analysis and data processing.

[0140] Based on the first or second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first or second embodiment can be referred to the above introduction, and will not be repeated in the following. Step S40 also includes steps D10 to D30:

[0141] Step D10, determining the data volume of the polygon data, and judging whether the memory resources of the computing node are less than a preset storage threshold;

[0142] It should be noted that a computing node refers to a computer or server node that participates in data processing or computing. Each computing node can process at least one task. In this embodiment, there are multiple computing nodes, which usually work together in a network environment. Memory resources refer to the memory space on a computing node that can be used to store and process data, including physical memory and virtual memory. The storage threshold is a preset upper limit value of the data volume, which is used to determine whether polygon data is suitable for direct storage in the memory of a computing node.

[0143] First, calculate the number of vertices of polygon data or the total number of data records to obtain the data volume. Then, check the memory resource usage of each computing node to determine whether each computing node has sufficient memory space to store the data. If there is a computing node whose memory resources are less than the storage threshold, it is determined that the memory resources of the computing node are insufficient. The evaluation of data volume and memory resources is the basis for determining the data storage method. By comparing the data volume and storage threshold, and checking the usage of memory resources, a reasonable storage method can be selected to avoid memory overflow or resource waste.

[0144] Step D20, if the amount of polygon data is less than the storage threshold and the memory resources of the computing node are sufficient, then determining the storage method is to store the polygon data in a preset total server memory, wherein the polygon data will be broadcast and saved to each computing node before calculation;

[0145] It should be noted that total server memory refers to the server JVM memory, which is usually used to store global data or shared data. Broadcasting is the process of copying data from one node to all other nodes in a network environment, ensuring that each node has a copy of the data.

[0146] If the data volume is less than the storage threshold and memory resources are sufficient, choose to store the polygon data in the total server memory. Before calculation, the polygon data is copied to all computing nodes through the broadcast mechanism to ensure that each node has a copy of the data. When the data volume is small and memory resources are sufficient, storing the data in the total server memory and copying it to each computing node through the broadcast mechanism can improve data access speed and computing efficiency. This is because memory access speed is much faster than disk access speed, and the broadcast mechanism can ensure data consistency and availability.

[0147] For example, refer to Figure 8 , Figure 8 This is a schematic diagram of broadcast variables. For small and medium-sized geographic data sets, the rasterized polygon data is fully stored in the server JVM memory of the terminal device. Before calculation, the model data is broadcast to each computing node, and each computing node saves a copy for multiple parallel tasks to call.

[0148] Step D30, if the polygon data is greater than or equal to the storage threshold or the memory resources of the computing node are insufficient, the storage method is determined to convert the polygon data into a resilient distributed dataset for storage, wherein the resilient distributed dataset has multiple storage points.

[0149] It should be noted that a resilient distributed dataset is a distributed data set that can process data in parallel across multiple computing nodes and has good fault tolerance and scalability. A storage point is a physical location or node in a distributed storage system where data is stored.

[0150] If the amount of data is greater than or equal to the storage threshold or memory resources are insufficient, choose to convert polygon data into a resilient distributed dataset (RDD) for storage and register it as a temporary table. When memory resources are insufficient, it can be overflowed to disk, which is more reliable. RDD has multiple storage points, which can store data in a dispersed manner to improve fault tolerance and scalability. When the amount of data is large or memory resources are insufficient, direct storage in memory may cause memory overflow or performance degradation. As a distributed data set, RDD can process data in parallel between multiple computing nodes to improve processing speed and fault tolerance. At the same time, RDD's multiple storage points can store data in a dispersed manner to reduce the storage pressure of a single node. When performing calculations and associations are required, you need to convert the longitude and latitude of the point into a raster index, associate a polygon object set with the temporary table, and then use a custom function to calculate the intersecting polygon object set.

[0151] This embodiment can flexibly select the storage method according to the amount of polygon data and the memory resources of the computing node. Different storage methods correspond to different data processing methods. The data stored in the main server memory can be quickly processed through simple memory access. The data stored in the RDD needs to use the parallel processing capabilities of the RDD for distributed computing. By selecting the appropriate storage method and processing method, data processing performance can be optimized and computing efficiency can be improved.

[0152] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the Spark-based geographic big data processing method of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0153] This application also provides a Spark-based geographic big data processing device, please refer to Fig. 9 , the Spark-based geographic big data processing device includes:

[0154] The parameter acquisition module 10 acquires a preset configuration file and performs configuration analysis on the configuration file to obtain input parameters;

[0155] The rasterization processing module 20 obtains geographic contour information according to input parameters, parses the geographic contour information into polygonal objects, and generates polygonal object raster data according to the polygonal objects in a preset raster system, wherein the raster system divides the polygonal objects into a grid structure, and one grid structure corresponds to one grid;

[0156] A data structure definition module 30 determines a data structure according to the data volume of the polygon object raster data, and converts the polygon object raster data into polygon data of the data structure, wherein the data structure includes a key-value pair and a double-layer key-value pair;

[0157] A storage mode determination module 40 determines a storage mode according to the amount of polygon data and the memory resources of a preset computing node, and stores the polygon data according to the storage mode;

[0158] The data processing module 50 runs the data processing command defined in the input parameters to process the polygon data, obtains output data, and outputs the output data.

[0159] The Spark-based geographic big data processing device provided by the present application adopts the Spark-based geographic big data processing method in the above-mentioned embodiment, which can solve the technical problem of low efficiency in geographic big data calculation. Compared with the prior art, the Spark-based geographic big data processing device provided by the present application has the same beneficial effects as the Spark-based geographic big data processing method provided by the above-mentioned embodiment, and other technical features in the Spark-based geographic big data processing device are the same as the features disclosed in the above-mentioned embodiment method, which will not be described in detail here.

[0160] The present application provides a Spark-based geographic big data processing device, and the Spark-based geographic big data processing device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Spark-based geographic big data processing method in the above-mentioned embodiment one.

[0161] Reference below Fig.10 , which shows a schematic diagram of the structure of a Spark-based geographic big data processing device suitable for implementing the embodiment of the present application. The Spark-based geographic big data processing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The Spark-based geographic big data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0162] like Fig.10 As shown, the Spark-based geographic big data processing device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of the Spark-based geographic big data processing device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the Spark-based geographic big data processing device to communicate wirelessly or wired with other devices to exchange data. Although the Spark-based geographic big data processing device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0163] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0164] The Spark-based geographic big data processing device provided by the present application adopts the Spark-based geographic big data processing method in the above-mentioned embodiment, which can solve the technical problem of low efficiency in geographic big data calculation. Compared with the prior art, the Spark-based geographic big data processing device provided by the present application has the same beneficial effects as the Spark-based geographic big data processing method provided in the above-mentioned embodiment, and the other technical features in the Spark-based geographic big data processing device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0165] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0166] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0167] The present application provides a medium, which is a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the Spark-based geographic big data processing method in the above-mentioned embodiment.

[0168] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0169] The above-mentioned computer-readable storage medium may be included in the Spark-based geographic big data processing device; or it may exist independently without being assembled into the Spark-based geographic big data processing device.

[0170] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the Spark-based geographic big data processing device, the Spark-based geographic big data processing device:

[0171] Get the preset configuration file, parse the configuration file, and get the input parameters;

[0172] Acquire geographic contour information according to input parameters, parse the geographic contour information into polygonal objects, and generate polygonal object raster data according to the polygonal objects in a preset raster system, wherein the raster system divides the polygonal objects into a grid structure, and one grid structure corresponds to one grid;

[0173] Determine a data structure according to the data volume of the polygon object raster data, and convert the polygon object raster data into polygon data of the data structure, wherein the data structure includes a key-value pair and a double-layer key-value pair;

[0174] Determine a storage method according to the amount of polygon data and the memory resources of a preset computing node, and store the polygon data according to the storage method;

[0175] Run the data processing command defined in the input parameters to process the polygon data, obtain output data, and output the output data.

[0176] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0177] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0178] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0179] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned Spark-based geographic big data processing method, and can solve the technical problem of low efficiency in geographic big data calculation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the Spark-based geographic big data processing method provided in the above-mentioned embodiment, and will not be repeated here.

[0180] The present application also provides a product, which is a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned Spark-based geographic big data processing method.

[0181] The computer program product provided by this application can solve the technical problem of low efficiency in computing geographic big data. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the Spark-based geographic big data processing method provided in the above embodiment, which will not be repeated here.

[0182] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.< / polygonintersectionitem> < / polygonintersectionitem> < / polygon> < / polygon> < / polygon>

Claims

1. A Spark-based geographic big data processing method, characterized in that: The Spark-based geographic big data processing method includes: Obtain a preset configuration file, and perform configuration analysis on the configuration file to obtain input parameters; Acquire geographic contour information according to the input parameters, parse the geographic contour information into polygonal objects, and generate polygonal object raster data according to the polygonal objects in a preset grid system, wherein the grid system divides the polygonal objects into a grid structure, and one grid structure corresponds to one grid; Determine a data structure according to the data volume of the polygon object raster data, and convert the polygon object raster data into polygon data of the data structure, wherein the data structure includes key-value pairs and double-layer key-value pairs; Determining a storage method according to the data volume of the polygon data and the memory resources of a preset computing node, and storing the polygon data according to the storage method; Run the data processing command defined in the input parameters to process the polygon data to obtain output data, and output the output data.

2. The Spark-based geographic big data processing method according to claim 1, characterized in that: The steps of acquiring geographic contour information according to the input parameters, parsing the geographic contour information into polygon objects, and generating polygon object raster data according to the polygon objects in a preset grid system include: Read the geographic contour information according to the data storage address in the input parameter, and parse the geographic contour information into the polygon object; Determine a bounding rectangle of each polygonal object, wherein the bounding rectangle is a minimum rectangle that contains the entire polygonal object; Performing intersection topology calculation on each grid in the circumscribed rectangle and the polygonal object through a preset grid system; The polygon object grid data is determined for the intersecting grids according to the intersection topology calculation result, wherein the polygon object grid data includes a data set of the polygon object and a data set of the grids intersecting with the polygon object.

3. The Spark-based geographic big data processing method according to claim 1, characterized in that: The step of determining the data structure according to the data volume of the polygon object grid data comprises: Determine the data volume of the polygon object raster data; If the data volume is less than a preset data volume threshold, the data structure is determined to be a key-value pair, wherein the key of the key-value pair includes a grid index value, and the value of the key-value pair is a data set of all polygon objects covered by the grid corresponding to the grid index value, wherein the grid index value is used to represent the position of the grid, including the grid side length, the grid longitude index value, and the grid latitude index value; If the data volume is greater than or equal to the data volume threshold, the data structure is determined to be a double-layer key-value pair, wherein the first layer key of the double-layer key-value pair is the grid longitude index value, the second layer key of the double-layer key-value pair is the grid latitude index value, and the value of the double-layer key-value pair is a sub-polygon object of the grid corresponding to the grid longitude index value and the grid latitude index value, and the sub-polygon object is the contour data of the intersecting part of each grid and the polygon object.

4. The Spark-based geographic big data processing method according to claim 3, characterized in that: Before the step of generating polygon object raster data according to the polygon object in the preset raster system, the method further includes: Determine a preset standard meridian length, determine a latitude difference per degree according to the standard meridian length, and determine a latitude difference per meter according to the latitude difference per degree, wherein the latitude difference per degree is an actual distance corresponding to each degree of latitude, and the latitude difference per meter is a latitude change corresponding to each meter; Determine a preset center point latitude value, determine a longitude difference per degree according to the center point latitude value and the latitude difference per degree, and determine a longitude difference per meter according to the longitude difference per degree, wherein the longitude difference per degree is the actual distance corresponding to each degree of longitude, and the longitude difference per meter is the longitude change corresponding to each meter; For each point data in the polygon object raster data, determine the longitude and latitude of the point data; Determine a normalized longitude code of the point data according to the longitude per meter difference and the longitude, wherein the normalized longitude code is used to characterize the position of the longitude in the grid system; Determine a latitude normalized code of the point data according to the latitude per meter difference and the latitude, wherein the latitude normalized code is used to characterize the position of the latitude in the grid system; The normalized longitude code is used as the grid longitude index value, the normalized latitude code is used as the latitude longitude index value, and the grid longitude index value and the latitude longitude index value are used as the grid index value.

5. The Spark-based geographic big data processing method according to claim 1, characterized in that: The step of determining the storage method according to the data volume of the polygon data and the memory resources of the preset computing node includes: Determining the data volume of the polygon data, and judging whether the memory resources of the computing node are less than a preset storage threshold; If the amount of the polygon data is less than the storage threshold and the memory resources of the computing node are sufficient, determining that the storage method is to store the polygon data in a preset total server memory, wherein the polygon data will be broadcast and saved to each computing node before calculation; If the polygon data is greater than or equal to the storage threshold or the memory resources of the computing node are insufficient, the storage method is determined to be to convert the polygon data into a resilient distributed dataset for storage, wherein the resilient distributed dataset has multiple storage points.

6. The Spark-based geographic big data processing method according to claim 1, characterized in that: The input parameters include user-defined function information, and the step of running the data processing command defined in the input parameters to process the polygon data to obtain output data includes: Register a custom function according to the custom function information, and bind the polygon data to the custom function; The step of running the data processing command defined in the input parameters to process the polygon data to obtain output data comprises: The user-defined function is called by executing the data processing command to process the polygon data and obtain output data.

7. A Spark-based geographic big data processing device, characterized in that: The Spark-based geographic big data processing device includes: The parameter acquisition module acquires a preset configuration file and performs configuration analysis on the configuration file to obtain input parameters; A rasterization processing module, which obtains geographic contour information according to the input parameters, parses the geographic contour information into polygonal objects, and generates polygonal object raster data according to the polygonal objects in a preset raster system, wherein the raster system divides the polygonal objects into a grid structure, and one grid structure corresponds to one grid; A data structure definition module, which determines a data structure according to the data volume of the polygon object raster data, and converts the polygon object raster data into polygon data of the data structure, wherein the data structure includes a key-value pair and a double-layer key-value pair; A storage mode determination module determines a storage mode according to the data volume of the polygon data and the memory resources of a preset computing node, and stores the polygon data according to the storage mode; The data processing module runs the data processing command defined in the input parameters to process the polygon data, obtains output data, and outputs the output data.

8. A Spark-based geographic big data processing device, characterized in that: The Spark-based geographic big data processing device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the Spark-based geographic big data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the Spark-based geographic big data processing method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the Spark-based geographic big data processing method as described in any one of claims 1 to 6.

Images