Method for supporting and optimally loading multi-source data of field work plotting

Abstract

The invention discloses a field work plotting multi-source data supporting and optimized loading method, particularly relates to the field of mobile geographic information systems, and realizes efficient plotting through multi-source data format compatibility supporting, dynamic loading optimization, memory and cache management and GPU rendering acceleration. A three-layer interface system is adopted to process heterogeneous data of vectors, grids, sensors and the like in a unified mode, coordinate system conversion is completed through a PROJ library, and the coordinate system is stored as a standardized GeoPackage; the dynamic loading of the view range is realized based on a quadtree index, and the loading efficiency is improved in combination with a multi-thread parallel processing and incremental updating mechanism; an improved LRU cache algorithm is adopted to optimize memory allocation, and vector data loading time is shortened through Zlib compression; a GPU vertex / fragment shader is used for realizing space coordinate conversion and Phong illumination rendering, elements with the same characteristic are combined into batch rendering tasks, and resource allocation is dynamically adjusted in combination with real-time performance monitoring.

Description

Technical Field

[0001] This invention relates to the field of mobile geographic information system technology, and more specifically, to a method for supporting and optimizing the loading of multi-source data for field mapping. Background Technology

[0002] With the development of information technology, mobile geographic information systems (GIS) are being used more and more widely in the field of field mapping, especially in the integration and loading of multi-source data.

[0003] Traditional field surveys rely on paper maps and manual recording, which is inefficient, prone to errors, and lacks data standardization and security.

[0004] With the advancement of digital transformation, field mapping is gradually moving towards electronic and information-based methods, adopting multiple data sources such as vector data, raster data, and remote sensing images. However, how to efficiently integrate data from different sources and achieve rapid loading and optimization remains a technical challenge.

[0005] Currently, although some field mapping systems manage data through geographic databases and support certain offline working modes, they have limitations in terms of compatibility with multi-source data, loading speed, and accuracy. Furthermore, issues related to data synchronization, updates, and redundancy remain unresolved. Therefore, a technical solution is needed that supports optimized loading of multi-source data to improve mapping efficiency and data quality, providing a more comprehensive solution for real-time processing and display. Summary of the Invention

[0006] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a method for supporting and optimizing the loading of multi-source data for field mapping, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: S1: Steps for compatibility support of multi-source data formats; Through a unified interface and data conversion module, it automatically identifies and processes multi-source heterogeneous data formats; S2: Optimized loading of mapping data; Data is dynamically loaded based on the user's current view scope, and is loaded in chunks and processed in parallel. S3: Memory management and cache optimization; A dynamic memory allocation strategy is adopted to allocate memory proportionally to the data cache, rendering buffer, and system reserved area. S4: Optimization of rendering of mapping data; GPU-accelerated rendering is achieved by using vertex and fragment shaders to enable vector symbolization and batch rendering.

[0008] Preferably, in step S1, the multi-source heterogeneous data formats include vector data, raster data database formats, service data, and sensor data; by integrating preprocessing tools to achieve standardized conversion of data formats and coordinate systems, the multi-source data is unified to a standard format and spatial reference, ensuring seamless compatibility and accurate spatial overlay of heterogeneous data, effectively supporting real-time rendering, spatial analysis, and other operations in the subsequent processing pipeline, and ultimately improving the system's data compatibility, coordinate accuracy, and business availability.

[0009] The unified interface design adopts a layered abstraction design, constructing a three-layer interface system, including a basic layer, an adaptation layer, and a service layer; The design adopts a three-layer interface system: the base layer defines a general interface for data access, including core functions such as metadata parsing, geometric data reading, and attribute table mapping; the adaptation layer implements specific adapters for different data formats to complete format parsing and conversion; the service layer dynamically loads adapters through the factory pattern, provides a unified call entry point, and realizes standardized output of multi-source data.

[0010] Preferably, in step S2, on-demand loading is optimized by the program algorithm based on the user's current view range, loading only the data within the current view range, thus reducing unnecessary data loading.

[0011] Chunking and parallel processing: Large-scale vector data is processed in chunks and loaded in parallel in a multi-threaded environment, which significantly improves loading efficiency. The parallel loading mechanism is as follows: In terms of hardware resource management, through: The `n_threads = max(4, Runtime.getRuntime().availableProcessors())` method retrieves the number of available CPU cores on the device in real time and sets a minimum number of threads to 4 to ensure basic performance. The final number of threads, `n_threads`, is the maximum of the number of cores and 4, ensuring optimal resource utilization on devices with varying performance levels. The task queue is implemented using a Linked Blocking Queue, with a task buffer of 100, effectively balancing the difference in task production and consumption rates and avoiding the risk of memory overflow.

[0012] In terms of task scheduling strategy, data areas are divided into three priority levels based on their importance: the currently visible area is given the highest priority of 1 to ensure that data within the user's field of view is loaded immediately; the pre-loaded area, usually a buffer area around the user's field of view, is given a medium priority of 0.5 to achieve a smooth roaming experience; and historical cached data is given a basic priority of 0.3 to optimize response speed in repeated access scenarios. The hierarchical strategy is generated through a weighted round-robin algorithm, which ensures that data in key areas is loaded first while also balancing overall performance.

[0013] Preferably, in step S3, the program designs a caching mechanism to store frequently used data in local memory or storage devices to avoid repeated loading and improve data access speed; based on the improved LRU cache eviction algorithm, combined with access frequency and time decay weights, cache resource management is optimized. The weighted formula for the improved LRU strategy in cache eviction algorithms is as follows: Cache score = 0.7 * access frequency weight + 0.3 * time decay weight; The access frequency weight is calculated by normalizing the number of the most recent N accesses using a sliding window, while the time decay weight decays exponentially based on the last access timestamp. When the cache space is insufficient, the data block with the lowest score is removed first. For preloaded data, a temporary protection period, such as 10 minutes, is set to avoid frequent eviction.

[0014] Preferably, in step S4, GPU-accelerated rendering, the symbolization and rendering tasks of vector data are handled by the GPU, which greatly improves graphics processing efficiency and reduces rendering latency and stuttering. A GPU-accelerated rendering pipeline was built, utilizing the parallel computing capabilities of mobile device graphics processors; the vertex shader is responsible for performing coordinate transformations and geometric calculations of spatial data, quickly converting geographic coordinates into screen coordinates; the fragment shader is used for symbolic rendering, supporting complex map style representations, including multi-level gradient color fills, texture mapping, dynamic lighting effects, etc. By using spatial indexing and feature classification, geographic features with the same rendering characteristics are intelligently grouped and merged into a single rendering batch; each batch can contain thousands of spatial features, significantly reducing the number of GPU rendering calls; at the same time, the system adopts advanced resource reuse technology to dynamically manage vertex buffers and texture resources.

[0015] The technical effects and advantages of this invention are as follows: This invention implements an optimized loading mechanism for multi-source data through a mobile application, significantly improving data loading speed and system response efficiency. By employing on-demand loading and resource allocation strategies, it effectively reduces lag during data loading and enhances data processing capabilities in large-scale field mapping tasks. Through optimized algorithms and storage mechanisms, it maintains high efficiency even under high loads, reducing memory usage and data processing latency. Simultaneously, improved rendering algorithms and dynamic adjustment strategies ensure stable display of multi-source data in different usage scenarios, thereby significantly improving the work efficiency and user experience of the field mapping system. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0017] Figure 2 This is a schematic diagram illustrating the program optimization principle of the present invention.

[0018] Figure 3 This is a schematic diagram illustrating the loading of full-scene image data into the field mapping system of the present invention.

[0019] Figure 4 A schematic diagram showing the loading of image data and standard 1:10000 scale map sheet vector data into the field mapping system of the present invention.

[0020] Figure 5 This is a schematic diagram illustrating the loading of massive contour vector data into the field mapping system of the present invention.

[0021] Figure 6 This is a schematic diagram illustrating the operation of the field mapping system of the present invention, which involves editing data from tens of thousands of nodes.

[0022] Figure 7 This is a test graph showing the monitoring of system memory changes during data loading according to the present invention. Detailed Implementation

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] Please see Figure 1 As shown, this invention provides a method for supporting and optimizing the loading of multi-source data for field mapping, including: S1: Steps for supporting compatibility of multi-source data formats; Automatically identify and process multi-source heterogeneous data formats through a unified interface and data conversion module; In step S1, the multi-source heterogeneous data formats include vector data, raster data database formats, service data, and sensor data. By integrating preprocessing tools, the data formats and coordinate systems are standardized and converted, unifying the multi-source data to a standard format and spatial reference. This ensures seamless compatibility and accurate spatial overlay of heterogeneous data, effectively supporting real-time rendering, spatial analysis, and other operations in the subsequent processing pipeline, ultimately improving the system's data compatibility, coordinate accuracy, and business availability.

[0025] The unified interface design adopts a layered abstraction design, constructing a three-layer interface system, including a basic layer, an adaptation layer, and a service layer; The design adopts a three-layer interface system: the base layer defines a general interface for data access, including core functions such as metadata parsing, geometric data reading, and attribute table mapping; the adaptation layer implements specific adapters for different data formats to complete format parsing and conversion; the service layer dynamically loads adapters through the factory pattern, provides a unified call entry point, and realizes standardized output of multi-source data. The data transformation process completes the standardization of multi-source data through the following steps: First, rigorous format checks are performed, ensuring data integrity through a dual verification mechanism of file header signature and extension. For example, Shapefile verifies the .shp / .shx / .dbf triples, and GeoJSON performs structural verification. Next, the open-source PROJ library is used to achieve high-precision coordinate system transformation, unifying various original coordinate systems to the CGCS2000 coordinate system with a transformation accuracy controlled within 0.1 meters of error. Then, data cleaning is performed, using the GDAL library to repair geometric topological errors, standardizing attribute field naming conventions, and setting a repair area threshold of 1e-6 square degrees. Finally, the processed data is written into a GeoPackage standardized container. Vector data is stored in the GPKG_GEOMETRY_COLUMNS table structure, and raster data is stored using the GPKG_TILE_MATRIX pyramid structure, generating spatial indexes and metadata records.

[0026] S2: Optimized data loading for plotting; dynamically loads data based on the user's current view range, and performs block loading and parallel processing; In step S2, on-demand loading is performed by optimizing the loading based on the user's current view range through program algorithms, loading only the data within the current view range, thus reducing unnecessary data loading.

[0027] Chunking and parallel processing: Large-scale vector data is processed in chunks and loaded in parallel in a multi-threaded environment, which significantly improves loading efficiency. The block processing uses a quadtree index, which is specifically: (1) Blocking rules: Maximum number of levels: 22 (corresponding to 0.3-meter resolution) Block size: 256x256 pixels (2) Index formula: Data block encoding: ZXY= (zoom_level, tile_x, tile_y); Tile X-coordinate calculation: tileX = floor( (longitude + 180) / 360 * 2^zoom_level ); Tile Y-coordinate calculation: tileY = floor([1 - (ln(tan(latitude*π / 180) + sec(latitude*π / 180))) / π] * 2^(zoom_level-1)); The parallel loading mechanism is as follows: In terms of hardware resource management, through: The `n_threads = max(4, Runtime.getRuntime().availableProcessors())` method retrieves the number of available CPU cores on the device in real time and sets a minimum number of threads to 4 to ensure basic performance. The final number of threads, `n_threads`, is the maximum of the number of cores and 4, ensuring optimal resource utilization on devices with varying performance levels. The task queue is implemented using a Linked Blocking Queue, with a task buffer of 100, effectively balancing the difference in task production and consumption rates and avoiding the risk of memory overflow.

[0028] In terms of task scheduling strategy, data areas are divided into three priority levels based on their importance: the currently visible area is given the highest priority of 1 to ensure that data within the user's field of view is loaded immediately; the pre-loaded area, usually a buffer area around the user's field of view, is given a medium priority of 0.5 to achieve a smooth roaming experience; and historical cached data is given a basic priority of 0.3 to optimize response speed in repeated access scenarios. The hierarchical strategy is generated through a weighted round-robin algorithm, which ensures that data in key areas is loaded first while also balancing overall performance.

[0029] The incremental loading mechanism detects changes in the view extent or adjustments in the zoom level. The system quickly calculates the range of newly added or changed data blocks using a spatial topology analysis algorithm and initiates a data request for the changed area. The specific implementation includes three steps: First, a spatial relationship model before and after the view change is established, and the difference area is quickly located using a quadtree index; then, the data blocks that need to be updated are identified based on the change detection algorithm; finally, the update task is broken down into atomic operations and added to the task queue.

[0030] S3: Memory management and cache optimization; adopts a dynamic memory allocation strategy to allocate memory proportionally to data cache, rendering buffer and system reserved area; In step S3, the program designs a caching mechanism to store frequently used data in local memory or storage devices to avoid repeated loading and improve data access speed; based on the improved LRU cache eviction algorithm, combined with access frequency and time decay weights, cache resource management is optimized. The weighting formula for the improved LRU strategy in cache eviction algorithms is as follows: Cache score = 0.7 * access frequency weight + 0.3 * time decay weight; The access frequency weight is calculated by normalizing the number of the most recent N accesses using a sliding window, while the time decay weight decays exponentially based on the last access timestamp. When the cache space is insufficient, the data block with the lowest score is removed first. For preloaded data, a temporary protection period, such as 10 minutes, is set to avoid frequent eviction. Memory optimization dynamically adjusts memory usage based on device performance, avoiding memory overflow issues; dynamic memory allocation specifically involves: Let the total memory threshold be M, and the proportion of each module be as follows: Data caching: 60%M, storing current view data and preloaded blocks, using memory pool technology to reduce fragmentation; Rendering buffer: 30%M, a reserved video memory mapping area for the GPU, supporting batch submission of rendering commands; System reserve: 10%M, to ensure basic operating system functions and emergency response; Memory usage is monitored via the JVM / native memory interface. If it exceeds the 80% threshold, garbage collection (GC) is immediately triggered and inactive cache is released. If the limit is continuously exceeded, data quality is gradually degraded.

[0031] Data compression: Lightweight compression technology is used for vector data during storage and loading, further reducing loading time; Vector data is compressed using Zlib (level 6), with a dictionary size of 32KB, a sliding window of 15 bits, and a compression rate of 50% to 70%.

[0032] S4: Optimize rendering of plotting data; accelerate rendering through GPU, and use vertex shaders and fragment shaders to achieve vector symbolization and batch rendering; In step S4, GPU accelerates rendering. The symbolization and rendering of vector data are handled by the GPU, which greatly improves graphics processing efficiency and reduces rendering latency and stuttering. A GPU-accelerated rendering pipeline was built, utilizing the parallel computing capabilities of mobile device graphics processors; the vertex shader is responsible for performing coordinate transformations and geometric calculations of spatial data, quickly converting geographic coordinates into screen coordinates; the fragment shader is used for symbolic rendering, supporting complex map style representations, including multi-level gradient color fills, texture mapping, dynamic lighting effects, etc. The transformation from geographic coordinates to screen coordinates is calculated using a composite transformation matrix. The specific calculation method is as follows: in, Represented as a projection matrix, Represented as a view matrix, Represented as a model matrix, Represented as homogeneous coordinates in the screen coordinate system. Represented as homogeneous coordinates in a geographic coordinate system; The Phong lighting model implemented in the fragment shader is as follows: in, Represented as the environmental reflectance coefficient, Expressed as diffuse reflectance coefficient, Let N represent the specular reflection coefficient, and N represent the normal vector. Indicated as the direction of the light source, Let I be the reflection vector, and let I be the total illumination intensity. Let V represent the ambient light intensity, and let V represent the viewing direction vector. This refers to the gloss level of a specular surface. By using spatial indexing and feature classification, geographic features with the same rendering characteristics are intelligently grouped and merged into a single rendering batch; each batch can contain thousands of spatial features, significantly reducing the number of GPU rendering calls; at the same time, the system adopts advanced resource reuse technology to dynamically manage vertex buffers and texture resources. The specific method for calculating the optimization objective function for batch processing of elements is as follows: in, This indicates whether element i is selected. This indicates that element i is selected. If , it means that element i is not selected; Represented as the importance weight of element i, This represents the maximum number of vertices in a single batch. Let the demand for the j-th resource be denoted as . Represented as elements The style attributes, Represented as the k-th group or set; The real-time performance monitoring module continuously tracks the load status of the rendering pipeline; when a performance bottleneck is detected, it automatically triggers dynamic optimization strategies, including adjusting the rendering detail level, intelligently allocating computing resources, and optimizing memory access patterns.

[0033] Please see Figure 7 The image shows a test graph for monitoring system memory changes during data loading. The test environment for monitoring system memory changes during data loading is as follows: Test device: HUAWEI MatePad Pro 12.6; Hardware configuration: Kirin 9000E / 8GB RAM / 256GB; Test dataset: 1:50,000 topographic map of China (2.1TB); Operating system: HarmonyOS 3.0; As shown in Table 1, Table 1 is a comparison table of the test boxes of the traditional solution and the solution of this solution; Table 1 Test Items Traditional solution This plan Improvement ratio Image loading speed 8.2s / GB 2.9s / GB 64.6%↑ Vector rendering delay 42ms 16ms 61.9%↓ Memory peak 1.8GB 1.1GB 38.9%↓ In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for supporting and optimizing the loading of multi-source data in field mapping, characterized in that, include: S1: Steps to adjust multi-source data format compatibility support; Through a unified interface and data conversion module, it automatically identifies and processes multi-source heterogeneous data formats; S2: Optimized loading of mapping data; Data is dynamically loaded based on the user's current view scope, and is loaded in chunks and processed in parallel. S3: Memory management and cache optimization; A dynamic memory allocation strategy is adopted to allocate memory proportionally to the data cache, rendering buffer, and system reserved area. S4: Optimized rendering of mapping data; GPU-accelerated rendering is achieved by using vertex and fragment shaders to enable vector symbolization and batch rendering.

2. The method for supporting and optimizing the loading of multi-source data for field mapping according to claim 1, characterized in that: In step S1, the multi-source heterogeneous data formats include vector data, raster data database formats, service data, and sensor data. By integrating preprocessing tools, the data formats and coordinate systems are standardized and converted, unifying the multi-source data to a standard format and spatial reference. This ensures seamless compatibility and accurate spatial overlay of heterogeneous data, effectively supporting real-time rendering, spatial analysis, and other operations in the subsequent processing pipeline, and improving the system's data compatibility, coordinate accuracy, and business availability. The unified interface design adopts a layered abstraction design, constructing a three-layer interface system, including the basic layer, the adaptation layer, and the service layer.

3. The method for supporting and optimizing the loading of multi-source data for field mapping according to claim 1, characterized in that: In step S2, data is loaded on demand based on the user's current view range, and only data within the current view range is loaded; block loading and parallel processing are used, and large-scale vector data is processed in blocks and loaded in parallel in a multi-threaded environment.

4. The method for supporting and optimizing the loading of multi-source data for field mapping according to claim 3, characterized in that: The block processing uses a quadtree index, which is specifically: Blocking rules: Maximum number of levels: 22 (corresponding to 0.3-meter resolution); Block size: 256x256 pixels; Index formula: Data block encoding: ZXY= (zoom_level, tile_x, tile_y); Tile X-coordinate calculation: tileX = floor( (longitude + 180) / 360 * 2^zoom_level ); Tile Y-coordinate calculation: tileY = floor([1 - (ln(tan(latitude*π / 180) + sec(latitude*π / 180))) / π] * 2^(zoom_level-1)).

5. The method for supporting and optimizing the loading of multi-source data for field mapping according to claim 3, characterized in that: The parallel loading mechanism is as follows: In terms of hardware resource management, through: `n_threads = max(4, Runtime.getRuntime().availableProcessors())` retrieves the number of available CPU cores on the device in real time and sets the minimum number of threads to 4 to ensure basic performance. The final number of threads, `n_threads`, is the maximum of the number of cores and 4, ensuring optimal resource utilization on devices with different performance levels. The task queue is implemented using a Linked Blocking Queue, with a task buffer of 100 to balance the difference in task production and consumption rates and avoid the risk of memory overflow.

6. The method for supporting and optimizing the loading of multi-source data for field mapping according to claim 1, characterized in that: In step S3, the program is designed with a caching mechanism to store frequently used data in local memory or storage devices. Based on the improved LRU cache eviction algorithm, and combined with access frequency and time decay weights, cache resource management is optimized; The weighting formula for the improved LRU strategy in cache eviction algorithms is as follows: Cache score = 0.7 * access frequency weight + 0.3 * time decay weight; The access frequency weight is calculated by normalizing the number of the most recent N accesses using a sliding window, while the time decay weight decays exponentially based on the last access timestamp. When the cache space is insufficient, the data block with the lowest score is removed first. For preloaded data, a temporary protection period is set.

7. The method for supporting and optimizing the loading of multi-source data for field mapping according to claim 1, characterized in that: In step S4, GPU-accelerated rendering is performed, and the symbolization and rendering tasks of vector data are handled by the GPU. Build a GPU-accelerated rendering pipeline to leverage the parallel computing capabilities of mobile device graphics processors; the vertex shader is responsible for performing coordinate transformations and geometric calculations of spatial data, quickly converting geographic coordinates into screen coordinates; Fragment shaders are used for symbolic rendering; By using spatial indexing and feature classification, geographic features with the same rendering characteristics are intelligently grouped and merged into a single rendering batch; each batch can contain thousands of spatial features, reducing the number of GPU rendering calls; at the same time, the system adopts resource reuse technology to dynamically manage vertex buffers and texture resources.

8. The method for supporting and optimizing the loading of multi-source data for field mapping according to claim 7, characterized in that: The transformation from geographic coordinates to screen coordinates is calculated using a composite transformation matrix. The specific calculation method is as follows: in, Represented as a projection matrix, Represented as a view matrix, Represented as a model matrix, Represented as homogeneous coordinates in the screen coordinate system. Represented as homogeneous coordinates in a geographic coordinate system; The Phong lighting model implemented in the fragment shader is as follows: in, Represented as environmental reflectance coefficient, Expressed as diffuse reflectance coefficient, Let N represent the specular reflection coefficient, and N represent the normal vector. Indicated as the direction of the light source, Let I be the reflection vector, and let I be the total illumination intensity. Let V represent the ambient light intensity, and let V represent the viewing direction vector. This refers to the gloss level of specular reflection.

Images