GIS map second-level loading method and system

Abstract

The invention belongs to the technical field of data processing, and particularly relates to a GIS map second-level loading method and system, and the system comprises an API gateway module which is used for receiving data query requests of a plurality of users, and transmitting all the data query requests; the data query module is used for constructing a geographic spatial index of the data novelty search request; the data search module is used for constructing a distributed index based on the geographic spatial index and carrying out data search on the stored geographic position information by utilizing the distributed index to obtain query geographic data; the data storage library is used for carrying out distributed storage on the geographical location information; the data processing module is used for performing window aggregation on the query geographic data to obtain aggregated data; the API gateway module is also used for distributing the plurality of aggregated data to the client; and the WebGL dynamic rendering module is used for converting the aggregated data into graphic objects in parallel by utilizing a GPU (Graphics Processing Unit) based on a WebGL technology to obtain a dynamic loading map.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a GIS map second-level loading method and system. BACKGROUND

[0002] As one of the core functions of the platform, GIS maps undertake the task of real-time display of key information such as station distribution, alarm status, personnel trajectory, and resource allocation.

[0003] In the prior art, a relational database (such as PostgreSQL, Oracle Spatial) is usually used to store geographic spatial data, and a spatial index (such as R-tree, quadtree) is used to speed up the query. When data is loaded, the system reads data from the database and renders it to the map. Distributed cache technology and distributed search engines are also used.

[0004] The prior art has the following technical problems: 1. In the emergency command and dispatching scenario, GIS maps need to load millions or even tens of millions of geographic spatial data (such as station information, alarm data, personnel trajectory, etc.). The existing GIS system significantly reduces the loading speed when processing such a large amount of data, and cannot meet the demand for second-level response.

[0005] 2. In emergency scenarios (such as typhoons, earthquakes, flood seasons, etc.), GIS maps need to real-time display of station status, personnel position, resource distribution, and other information in the affected area. The existing technology is difficult to realize real-time updating and rendering of data, resulting in decision delay.

[0006] 3. During major event protection or sudden event disposal, the platform may face a large number of concurrent accesses of users. The existing GIS system is prone to performance degradation or even crash in high-concurrency scenarios, affecting the efficiency of emergency command.

[0007] 4. Emergency command and dispatching not only need to display basic geographic information, but also need to support complex query and analysis functions (such as station fault statistics in the region, resource allocation optimization, etc.). The existing technology is inefficient in processing these complex queries, and is difficult to meet business needs.

[0008] 5. GIS maps need to real-time display dynamic data (such as personnel trajectory, resource movement, etc.). The existing technology has limited support for dynamic data, and it is difficult to achieve efficient visualization and interaction.

[0009] 6. Geographic spatial data usually has a large amount of data, and the existing technology has a large overhead in data transmission and storage, affecting system performance and resource utilization.

[0010] 7. Existing GIS systems typically employ a monolithic architecture, making horizontal scaling difficult and incurring high maintenance costs. As business demands grow, system scalability and maintainability become bottlenecks. Summary of the Invention

[0011] In view of the technical problems existing in the prior art, the present invention provides a GIS map loading method and system in seconds. Its purpose is to combine an innovative data loading algorithm with distributed caching, real-time search engine, data layer loading and dynamic compression technology to enable massive geospatial data to be loaded in seconds.

[0012] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: a GIS map loading system with a second-level loading capability, the system comprising: The API gateway module is used to receive data query requests from multiple users and transmit all data query requests. The data query module is used to receive all data novelty requests and build a geospatial index for the data novelty requests; The data search module is used to construct a distributed index based on the geospatial index, and use the distributed index to search for stored geographic location information to obtain the queried geographic data. A data repository for distributed storage of geographic location information; The data processing module is used to receive the queried geographic data in real time and perform window aggregation on the queried geographic data to obtain aggregated data; The API gateway module is also used to distribute multiple aggregated data to clients; The WebGL dynamic rendering module is used to convert aggregated data into graphical objects in parallel using the GPU based on WebGL technology, resulting in dynamically loaded maps.

[0013] Furthermore, the above also includes: The interactive map control module is used to update the current view of the dynamically loaded map in real time based on user interaction.

[0014] Furthermore, the API gateway module described above includes: The load balancing unit is used to distribute user data query requests to microservers through reverse proxy and load balancing methods. The authentication and rate limiting unit is used to verify the user's identity information and, based on the user's identity information, to limit the user's high-concurrency data query requests using the token bucket method.

[0015] Furthermore, the data query module described above includes: The Elasticsearch query unit is configured to receive a data query request of a user, and generate a geospatial index according to the data query request; wherein the geospatial index supports range query and proximity query. The Redis cache interaction unit is configured to cache the aggregated data meeting preset conditions by using an LRU method; wherein the preset conditions are that the aggregated data appears more than a preset number of times within a preset time.

[0016] Further, the data search module includes: The geospatial index unit is configured to construct a multi-level spatial index by using geo_point and geo_shape field types in the geospatial index, and perform data search on the stored geographic location information by using the multi-level spatial index to obtain the queried geographic data. The shard and replica mechanism unit is configured to store the geographic location information in shards in a plurality of nodes, and set replicas of the geographic location information.

[0017] Further, the data storage library includes a distributed cache module, and specifically includes: The data structure optimization unit is configured to store the geographic location information by using a Redis GEO module. The cache preheating unit is configured to preheat the hot data in the geographic location information in a timely manner.

[0018] Further, the data storage library further includes: The relational database module is configured to store static data.

[0019] Further, the data processing module includes: The Kafka message queue consumption unit is configured to receive the queried geographic data in real time, and process the received queried geographic data by using a partition mechanism and a consumer group method. The Flink stream computing unit is configured to perform window aggregation on the processed queried geographic data to obtain the aggregated data.

[0020] Further, the WebGL dynamic rendering module includes: The WebGL rendering engine unit is configured to convert the aggregated data into vertex data, and render the vertex data by using a shader to obtain a dynamically loaded map; wherein the vertex data is stored in a buffer. The GPU accelerated computing unit is configured to perform parallel processing on the conversion process of converting the aggregated data into vertex data by using a Compute Shader method of WebGL.

[0021] Secondly, to solve the above-mentioned technical problems, the present invention also provides a GIS map loading method with a second-level loading capability, used in the aforementioned GIS map loading system with a second-level loading capability, the method comprising: Receive data query requests from multiple users; Based on the data novelty search request, construct a geospatial index of the data novelty search request; A distributed index is constructed based on the geospatial index, and the distributed index is used to search for stored geographic location information to obtain the queried geographic data. The queried geographic data is then aggregated using a window to obtain aggregated data. Based on WebGL technology, the aggregated data is converted into graphical objects in parallel using the GPU to obtain a dynamically loaded map.

[0022] Compared with the prior art, the present invention has the following advantages: 1. This invention achieves sub-second query speeds for massive datasets by utilizing Redis caching and Elasticsearch distributed indexing. WebGL and GPU acceleration technologies improve rendering performance by more than 10 times. It successfully solves the core challenges faced by traditional GIS systems in handling massive data loading, achieving real-time response, and dealing with high-concurrency scenarios.

[0023] 2. In terms of emergency command and dispatch, this invention can overlay typhoon paths onto a map in real time, displaying the number of station failures and resource distribution in affected areas within seconds, providing decision-makers with immediate and accurate information support. Even more conveniently, the system supports one-click generation of emergency plans, greatly improving the speed and efficiency of emergency response.

[0024] 3. In the context of major event security, this invention can monitor the movement of personnel and the status of resources within the security area in real time. By analyzing densely populated areas using heat maps, it provides security personnel with an intuitive and dynamic overview of the situation on-site. Based on this, security strategies can be adjusted quickly and flexibly to ensure the safe and smooth conduct of the event.

[0025] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0027] Figure 1 A structural schematic diagram of a GIS map second-level loading system according to an embodiment of the present application is shown. Figure 2 A flowchart of a GIS map second-level loading method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely explain the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0029] Figure 1 A structural schematic diagram of a GIS map second-level loading system according to an embodiment of the present application is shown. Figure 1 As shown in the figure, the embodiment of the present application includes a user interaction layer, an API gateway layer, a micro-service application layer and a data storage layer, and specifically includes: An API gateway module is used to receive data query requests of multiple users and transmit all the data query requests. A data query module is used to receive all the data query requests and build a geographic spatial index of the data query requests. A data search module is used to build a distributed index based on the geographic spatial index and use the distributed index to search the stored geographic position information to obtain query geographic data. A data storage library is used to store the geographic position information in a distributed manner. A data processing module is used to receive the query geographic data in real time and perform window aggregation on the query geographic data to obtain aggregated data. The API gateway module is also used to distribute the multiple aggregated data to the client. A WebGL dynamic rendering module is used to convert the aggregated data into graphic objects based on WebGL technology using GPU parallelism to obtain a dynamically loaded map.

[0030] In this embodiment, the microservice architecture is adopted to split the system functions into independent service modules, supporting horizontal expansion and high concurrency processing. The asynchronous processing mechanism is introduced to execute time-consuming operations in the background, improving the smoothness of front-end interaction.

[0031] Real-time load based on service nodes: CPU, memory, network delay, calculate weight value:

[0032] wherein, represents the weight value, represents the CPU usage, represents the memory usage, represents the network delay.

[0033] Optionally, it further includes: An interactive map control module is used to update the current view of the dynamically loaded map in real time according to the user's interactive operation.

[0034] In this embodiment, the data layer loading technology is used to load only the dynamically loaded map within the current view range, reducing the amount of data loaded at one time. The dynamic compression technology is applied to further reduce the data transmission amount and improve the loading efficiency. The dynamic compression algorithm triggers the condition when the data transmission amount > 10MB or the network delay > 100ms; Optionally, the API gateway module includes: A load balancing unit is used to distribute the user's data query request to the micro server through the reverse proxy method and the load balancing method; An authentication and flow limiting unit is used to verify the user's identity information, and based on the user's identity information, the user's high-concurrency data query request is limited by using the token bucket method.

[0035] In this embodiment, OAuth2.0 / JWT is integrated to realize user identity authentication, and the API gateway module supports dynamic routing configuration, which can flexibly cope with the traffic demand of different business scenarios. Combined with the microservice governance tool, service fusing and degradation are realized.

[0036] Optionally, the data query module includes: An Elasticsearch query unit is used to receive the user's data query request and generate a geographic spatial index according to the data query request; wherein the geographic spatial index supports range query and proximity query; A Redis cache interaction unit is used to cache the aggregated data that meets the preset condition through the LRU method; wherein the preset condition is that the aggregated data appears more than a preset number of times within a preset time.

[0037] In this embodiment, by caching high-frequency aggregated data, the aggregated data is directly output according to the corresponding data query request, and database access delay is reduced.

[0038] Optionally, the data search module comprises: The geospatial index unit is configured to construct a multi-level spatial index through the geo_point and geo_shape field types in the geospatial index, and perform data search on the stored geographic location information through the multi-level spatial index to obtain the queried geographic data. The sharding and replication mechanism unit is configured to store the geographic location information in sharding manner in a plurality of nodes and set a replica of the geographic location information.

[0039] In this embodiment, the sharding and replication mechanism unit supports horizontal expansion by guaranteeing high availability through replication.

[0040] In this embodiment, a hybrid index structure of GeoHash+inverted index is adopted: the geographic coordinates are converted into GeoHash codes to establish a prefix tree index; inverted indexes are established for the business attribute fields; the number of index shards is dynamically adjusted through an adaptive sharding algorithm, and the formula is: number of shards = max(total data volume / 5GB, number of nodes x 2); and then a real-time query routing algorithm is used to intelligently route the query requests.

[0041] Optionally, the data storage library comprises a distributed cache module, and specifically comprises: The data structure optimization unit is configured to store the geographic location information using the Redis GEO module. The cache preheating unit is configured to periodically preheat the hotspot data in the geographic location information.

[0042] Optionally, the data storage library further comprises: The relational database module is configured to store static data.

[0043] In this embodiment, the distributed cache module is used to store system cache hotspot data, thereby reducing the database query pressure. In combination with a distributed search engine such as Elasticsearch, fast query and analysis of massive geospatial data are realized. Specifically: The data is subjected to heat analysis: based on historical access frequency and spatiotemporal correlation, the hotspot data in the geographic location information is dynamically identified through a sliding window method, and the formula is: heat weight = access frequency x time decay factor (decay coefficient = 0.8 / minute).

[0044] A three-level Redis cluster cache system is constructed: L1 cache: stores the vector data within the current view range, and TTL = 5 minutes. L2 cache: store the aggregation result of high frequency query, TTL=15 minutes; L3 cache: store the pre-computation result of Elasticsearch index; Hot data pre-warming mechanism: based on user behavior prediction, asynchronously preload adjacent view data through background thread.

[0045] Optionally, the data processing module comprises: A Kafka message queue consumption unit, configured to receive query geographic data in real time, and process the received query geographic data through a partition mechanism and a consumer group method; A Flink stream computing unit, configured to perform window aggregation on the processed query geographic data to obtain aggregated data.

[0046] Optionally, the WebGL dynamic rendering module comprises: A WebGL rendering engine unit, configured to convert the aggregated data into vertex data, and render the vertex data by using a shader to obtain a dynamically loaded map; wherein the vertex data is stored in a buffer; A GPU accelerated computing unit, configured to perform parallel processing on the conversion process of converting the aggregated data into vertex data by using a Compute Shader method of WebGL.

[0047] In this embodiment, the WebGL rendering engine unit supports real-time drawing of massive point, line and surface data. Through GPU parallel computing capability, the aggregated data is converted into a graphic object to realize second-level rendering. The data within the current view range is dynamically loaded to reduce the front-end memory occupation.

[0048] In this embodiment, converting the aggregated data into vertex data specifically comprises: compressing the vertex data, and converting geographic coordinates into a Float32Array format recognizable by WebGL; and writing a GLSL shader to render the vertex data.

[0049] In this embodiment, the double-buffering technology is used to realize a non-stuttering rendering process, which can separate display from data preparation to improve rendering efficiency and user experience. Specifically, the front end is responsible for displaying the buffer used in the current frame to ensure that the user sees the completely prepared picture content; meanwhile, the background uses a Web Worker thread to asynchronously prepare data for the next frame.

[0050] Based on the same principle as the method shown in Figure 1 The embodiment of the application also provides a GIS map second-level loading method, as shown in Figure 2 The method comprises the following steps: receiving data query requests of a plurality of users; constructing a geospatial index of the data novelty request based on the data novelty request; constructing a distributed index based on the geospatial index, and performing data search on the stored geolocation information by using the distributed index to obtain query geographic data; performing window aggregation on the query geographic data to obtain aggregated data; based on WebGL technology, converting the aggregated data into graphic objects by using GPU in parallel to obtain a dynamically loaded map.

[0051] The GIS map second-level loading method provided by the embodiment of the present application can execute the GIS map second-level loading system provided by the embodiment of the present application, and the implementation principle is similar, which will not be repeated here.

[0052] The GIS map second-level loading system can be a computer program (including program code) running in a computer device, for example, the GIS map second-level loading system is an application software; the application software can be used to execute the corresponding steps in the method provided by the embodiment of the present application.

[0053] The modules described in the embodiments of the present application can be realized by software or hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0054] The above description is only the preferred embodiment of the present application and the explanation of the applied technical principles. Those skilled in the art should understand that the disclosed range in the present application is not limited to the technical solution formed by the specific combination of the above technical features, and also covers other technical solutions formed by any combination of the above technical features or equivalent features without departing from the disclosed concept. For example, the above features are replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form a technical solution.

Claims

1. A GIS map second-level loading system, characterized in that, The system comprises: An API gateway module for receiving data query requests of multiple users and transmitting all data query requests; A data query module for receiving all data query requests and constructing a geospatial index of the data query requests; A data search module for constructing a distributed index based on the geospatial index and searching data stored in the geolocation information based on the distributed index to obtain query geographic data; A data storage library for storing the geolocation information in a distributed manner; A data processing module for receiving the query geographic data in real time and performing window aggregation on the query geographic data to obtain aggregated data; The API gateway module is further configured to distribute the aggregated data to a client; A WebGL dynamic rendering module for converting the aggregated data into graphic objects based on WebGL technology and GPU parallelism to obtain a dynamically loaded map.

2. The GIS map second-level loading system according to claim 1, characterized in that, Further comprising: An interactive map control module for updating a current view of the dynamically loaded map in real time according to user interactions.

3. The GIS map second-level loading system according to claim 1, characterized in that, The API gateway module comprises: A load balancing unit for distributing data query requests of users to micro servers through a reverse proxy method and a load balancing method; An authentication and flow limiting unit for verifying identity information of users and limiting high-concurrency data query requests of users based on the identity information of the users using a token bucket method.

4. The GIS map second-level loading system according to claim 1, characterized in that, The data query module comprises: An Elasticsearch query unit for receiving data query requests of users, generating a geospatial index based on the data query requests; wherein the geospatial index supports range queries and proximity queries; A Redis cache interaction unit for caching aggregated data meeting preset conditions through an LRU method; wherein the preset conditions are that the aggregated data appears more than a preset number of times within a preset time.

5. The GIS map second-level loading system according to claim 1, characterized in that, The data search module comprises: A geospatial index unit for constructing a multi-level spatial index through geo_point and geo_shape field types in the geospatial index, searching data stored in the geolocation information through the multi-level spatial index, and obtaining query geographic data; A sharding and replication mechanism unit for sharding and storing the geolocation information in multiple nodes and setting replicas of the geolocation information.

6. The GIS map second-level loading system according to claim 5, characterized in that, The data storage library comprises a distributed cache module, specifically comprising: A data structure optimization unit for storing the geolocation information using a Redis GEO module; A cache preheating unit for preheating hot data in the geolocation information at regular intervals.

7. The GIS map second-level loading system according to claim 6, characterized in that, The data storage library further comprises: A relational database module for storing static data.

8. The GIS map second-level loading system according to claim 1, characterized in that, The data processing module comprises: A Kafka message queue consumption unit for receiving query geographic data in real time and processing the received query geographic data through a partition mechanism and a consumer group method; A Flink stream computing unit for performing window aggregation on the processed query geographic data to obtain aggregated data.

9. The GIS map second-level loading system according to claim 1, characterized in that, The WebGL dynamic rendering module comprises: The WebGL rendering engine unit is configured to convert the aggregated data into vertex data and render the vertex data by using a shader to obtain a dynamically loaded map. The vertex data is stored in a buffer. The GPU acceleration computing unit is configured to perform a conversion process of converting the aggregated data into vertex data in parallel by using a Compute Shader method of WebGL.

10. A GIS map second-level loading method, used for the GIS map second-level loading system in any of claims 1-9, characterized in that, The method comprises: receiving data query requests of a plurality of users; constructing a geographic spatial index of the data query requests based on the data query requests; constructing a distributed index based on the geographic spatial index, and performing data search on stored geographic position information by using the distributed index to obtain query geographic data; performing window aggregation on the query geographic data to obtain aggregated data; converting the aggregated data into a graphic object in parallel by using a GPU based on a WebGL technology to obtain a dynamically loaded map.

Images