A lightweight multi-source GIS data management method based on sidecar mode

Abstract

The application relates to the technical field of computer software, and discloses a lightweight multi-source GIS data management method based on a Sidecar mode, which receives and analyzes uploaded GIS data files; generates metadata files stored in the same directory based on the Sidecar mode; when data needs to be displayed on a Web map, performs coordinate conversion on a server according to the metadata; when a vector data preview request is received, performs real-time streaming analysis and dynamic conversion into a front-end renderable format for return; and simultaneously monitors storage status and performs pre-judgment of space before uploading. The system adopts a front-end and back-end separation architecture to realize the above method. Through physical binding of metadata and entity files, the application solves the synchronization problem; through server-side coordinate conversion, the application overcomes the defect that the Web end is insufficiently supported in complex projection; and through streaming analysis of vector data, the application avoids storage redundancy and realizes lightweight, low-cost and strongly consistent GIS data management.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, and in particular to a lightweight multi-source GIS data management method based on the Sidecar pattern. Background Technology

[0002] With the deepening application of remote sensing technology and geographic information, departments such as natural resources, geological surveys, and ecological and environmental protection have accumulated massive amounts of geospatial data. These data come from a wide range of sources and are diverse in format (such as satellite imagery GeoTIFF, vector data Shapefile, UAV orthophotos DOM, etc.), and are usually characterized by large data volume (GB level) and complex coordinate systems.

[0003] Existing GIS data management systems typically employ the following technical architecture: The "file + database" model stores entity files on disk and metadata (filename, path, spatial range, etc.) in a relational database (such as MySQL, Oracle) or a spatial database (such as PostgreSQL / PostGIS).

[0004] Dependence on heavy middleware: Publishing map services typically relies on professional GIS server software such as GeoServer and ArcGIS Server.

[0005] The shortcomings of existing technology are as follows: High deployment and maintenance costs: It requires the installation and configuration of complex databases and GIS servers, the environment setup is cumbersome, and the hardware resource requirements are high, making it unsuitable for lightweight application scenarios or edge computing environments.

[0006] Metadata synchronization difficulties: When files are moved, renamed, or deleted directly in the file system, the records in the database often cannot be updated in time, resulting in "dead links" or data inconsistency.

[0007] Poor compatibility with web-based projection: Web maps typically use WGS84 or Web Mercator coordinate systems. Existing front-end transformation libraries (such as proj4js) have limited support for complex projections (especially custom local projections generated by drones), often resulting in positional offsets or loading failures when data is overlaid.

[0008] Large storage redundancy: In order to enable web preview of vector data such as Shapefile, it is usually necessary to convert it into GeoJSON format file in advance for storage, resulting in double the storage space usage. Summary of the Invention

[0009] The purpose of this invention is to propose a lightweight multi-source GIS data management method based on the Sidecar pattern, which aims to solve the synchronization difficulties caused by the separation of metadata and entity files.

[0010] Specifically, this invention provides a lightweight multi-source GIS data management method based on the Sidecar pattern, which includes the following steps: S1. Receive the GIS data file uploaded by the user, and parse the spatial information and attribute information of the GIS data file through the backend GDAL engine; S2. Generate a metadata file corresponding to the GIS data file based on the Sidecar pattern, and store the metadata file and the GIS data file in the same directory; S3. When it is necessary to display the GIS data on a Web map, the spatial range of the data is transformed according to the coordinate projection information in the metadata file to unify it into a coordinate system compatible with the Web map. S4. When a preview request for vector format GIS data is received, the corresponding binary vector file is parsed in real time, dynamically converted into a format that can be rendered by the front end, and returned. S5. Monitor the system storage status and perform space prediction before uploading.

[0011] A lightweight multi-source GIS data management system based on the Sidecar pattern, employing a front-end / back-end separated B / S architecture, includes: The front-end presentation layer is used for map rendering, user interaction, and data requests. The backend business logic layer is used to execute steps S1 to S5. The data storage layer is used to store GIS data files and their corresponding Sidecar metadata files.

[0012] The beneficial effects provided by this invention are: 1. Extremely low deployment and maintenance costs: It abandons traditional relational databases and heavy GIS server middleware (such as GeoServer). The system only requires a Node.js runtime environment for deployment, achieving true "out-of-the-box" functionality. This greatly reduces the software and hardware threshold and is particularly suitable for lightweight applications, rapid prototyping, and edge computing scenarios.

[0013] 2. Strong consistency in metadata management: The innovative application of the Sidecar pattern from microservices to GIS metadata management allows metadata files (.meta.json) to physically coexist and move together with the original data files. This fundamentally solves the problem of "dead links" or inconsistencies in metadata records caused by file system operations in the traditional "file + database" model, ensuring data integrity and portability.

[0014] 3. Achieved full coordinate system compatibility and accurate overlay: Addressing the industry pain point of displaying custom projection data such as drone imagery with offsets on the web, this invention utilizes the GDAL engine on the server side for intelligent projection recognition and forced coordinate transformation (converting the data range to WGS84). This process is completed entirely before front-end rendering, ensuring that data from any coordinate system can achieve accurate spatial overlay on standard web map bases such as Web Mercator, thus improving the accuracy of data fusion.

[0015] 4. High storage efficiency and no redundant conversion: For vector data such as Shapefiles, the conventional approach of "pre-converting and storing upon upload" is abandoned, and a strategy of "real-time streaming parsing during preview" is adopted. Binary data is dynamically converted to formats such as GeoJSON in memory only when responding to frontend requests, and then released after use. This eliminates the storage space waste (usually 100% redundancy) caused by storing an extra copy of the converted file for preview, significantly improving storage efficiency. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is the Sidecar generation flowchart; Figure 3 This is a flowchart of coordinate transformation; Figure 4 This is a schematic diagram of the system architecture of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0018] Before formally describing the present invention, a general description of the solution of the present invention will be given first to facilitate understanding.

[0019] Example 1 Please refer to Figure 1 This invention provides a lightweight multi-source GIS data management method based on the Sidecar pattern, comprising the following steps: S1. Receive the GIS data file uploaded by the user, and parse the spatial information and attribute information of the GIS data file through the backend GDAL engine; Please refer to Figure 2 , Figure 2 This is the Sidecar generation flowchart; Specifically, when a user uploads a GIS data file (such as `data.tif`), the system backend automatically calls the GDAL engine to parse the spatial information (range, resolution, projection) and attribute information of the file.

[0020] S2. Generate a metadata file corresponding to the GIS data file based on the Sidecar pattern, and store the metadata file and the GIS data file in the same directory; It should be noted that step S2 includes the following sub-steps: S21. Based on the parsing results, create a metadata file in .meta.json format with the same name as the GIS data file in the same directory; S22. Write the original name, size, modification time, spatial range, and coordinate projection information of the file into the metadata file; Specifically, the metadata file records structured information such as the original name, size, modified time, bounds, and projection code of the data.

[0021] S23. In response to the file list request, read multiple .meta.json files in the directory and assemble them into a list and return it.

[0022] Specifically, when the front-end requests a list of files, the system concurrently reads all `.meta.json` files in the directory, assembles them directly into a JSON list, and returns it. This approach avoids the I / O bottleneck of database queries and allows data backup and migration to be completed simply by copying folders.

[0023] It should be noted that after the front-end map component loads the data list, when the user clicks the 'Preview' or 'Load to Map' button for a data item, the front-end immediately sends two parallel API requests to the back-end: one to obtain the spatial extent information of the data (for map positioning), and the other to request the data content itself (for rendering). For the first request, the back-end executes step S3; for the second request, if it is in vector format, it executes step S4; if it is in raster format, it directly provides the file access path or tile service.

[0024] S3. When it is necessary to display the GIS data on a Web map, the spatial range of the data is transformed according to the coordinate projection information in the metadata file to unify it into a coordinate system compatible with the Web map. Please refer to Figure 3 , Figure 3 This is a flowchart of coordinate transformation; It should be noted that step S3 includes the following sub-steps: S31. Read coordinate projection information from the metadata file; Specifically, the first step is to use GDAL's C++ low-level interface to read the WKT (Well-KnownText) projection definition string that comes with the data.

[0025] S32. If the projection information is a standard EPSG code, then it shall be used directly; Specifically, if the WKT contains a standard EPSG authorization code, extract and use that EPSG code directly (e.g., `EPSG:32650`).

[0026] If no standard code is available (such as a custom projection generated by a drone), the system automatically analyzes the WKT content. If keywords such as "UTM zone" are detected, the corresponding standard UTM projection code is automatically derived.

[0027] S33. If the projection information is a custom projection definition, the spatial range of the data is forcibly converted from the source coordinate system to the WGS84 coordinate system through the GDAL engine.

[0028] Specifically, for completely custom or unknown projections, the system uses GDAL's `CoordinateTransformation` interface on the server side to directly convert the data's boundary coordinates (Envelop) from the source coordinate system to WGS84 latitude and longitude coordinates. This process bypasses the limitations of the front-end `proj4js` library's insufficient support for complex projections, ensuring accurate overlay of all data on the web map.

[0029] It should be noted that in step S33, the forced conversion is completed on the server side, and the conversion result is only the spatial range coordinates for front-end positioning, without modifying the original data file.

[0030] The 'spatial extent' specifically refers to the geographic coordinate bounding box read from the metadata file or raw data, typically represented by minimum / maximum latitude and longitude (or projected coordinates). Forced transformation utilizes GDAL's CoordinateTransformation function to convert the coordinates of the four corner points of this bounding box from the source projection to WGS84 latitude and longitude coordinates. The transformation result (a set of WGS84 coordinates) is encapsulated in a JSON response and returned to the front end. The front-end map library (such as Leaflet) directly uses this extent for map view positioning and zooming without requiring any projection calculations on the front end.

[0031] S4. When a preview request for vector format GIS data is received, the corresponding binary vector file is parsed in real time, dynamically converted into a format that can be rendered by the front end, and returned. It should be noted that step S4 includes the following sub-steps: S41. Verify the integrity of the requested vector data file; Specifically, to enable online preview of binary vector data such as Shapefiles, the system adopts a streaming parsing scheme: First, during the upload stage, the system automatically detects the necessary file combination of Shapefile (`.shp`, `.shx`, `.dbf`), none of which can be missing.

[0032] S42. Real-time reading of the geometric and attribute binary streams of the vector data; Secondly, when the front end requests a preview, the back end does not perform pre-conversion of the data format (such as saving it as a GeoJSON file), but instead reads the `.shp` and `.dbf` binary streams in real time.

[0033] S43. Dynamically parse and concatenate the data into a GeoJSON format data stream in memory; It should be noted that the dynamically generated GeoJSON format data in step S43 is only for this preview and will not be persistently stored.

[0034] The aforementioned 'dynamic parsing and concatenation into a GeoJSON format data stream' refers to the backend using shapefiles or similar libraries to read features one by one (or in batches) from .shp and .dbf files. After parsing each feature, it is immediately formatted into a JSON string fragment conforming to the GeoJSONFeature object specification and progressively sent to the frontend via an HTTP response stream. The frontend can use APIs such as Fetch to read the streaming data and use JSON.parse to parse the arriving fragments, enabling data to be transmitted, parsed, and rendered simultaneously, improving the response speed and user experience of large file previews.

[0035] S44. Return the data stream to the front end for rendering.

[0036] Finally, the geometry and attribute data are parsed using the `shapefile` library and dynamically concatenated into a GeoJSON object stream, which is then returned to the front end. This "use-as-you-go" strategy saves a significant amount of server storage space and ensures data real-time performance.

[0037] S5. Monitor the system storage status and perform space prediction before uploading.

[0038] Specifically, the present invention has intelligent storage monitoring function, as follows: Cross-platform detection: Automatically identifies the operating environment (Windows / Linux). On Windows, it calls the `fs.statfs` or `wmic` interface; on Linux, it calls the `df` command.

[0039] Real-time calculation: Get the total capacity, used space, and remaining space of the disk partition where the project is located in real time.

[0040] Dynamic feedback: Disk usage is fed back to the front end in real time via API, and space is predicted before uploading large files to prevent data corruption caused by disk overflow.

[0041] As one example, during system startup, the operating system is determined via `process.platform`. In a Linux environment, the `df -k / path / to / storage` command is executed using `child_process.exec`, and the output is parsed to obtain disk block information. In a Windows environment, information can be obtained using third-party NPM libraries such as `node-df` or by executing the command `wmic logicaldiskwhere caption="C:" get size,freespace`. The obtained raw data is then processed and provided to the front end through a separate monitoring API interface.

[0042] Example 2: Please refer to Figure 4 , Figure 4 This is a schematic diagram of the system architecture of the present invention.

[0043] A lightweight multi-source GIS data management system based on the Sidecar pattern, employing a front-end / back-end separated B / S architecture, includes: The front-end presentation layer is used for map rendering, user interaction, and data requests. The backend business logic layer is used to execute steps S1 to S5. The data storage layer is used to store GIS data files and their corresponding Sidecar metadata files.

[0044] As one example, the frontend is built on the React 18 and Next.js 14 frameworks. Server-Side Components (RSC) technology is used to improve the performance of the first screen rendering; Leaflet.js is used as a lightweight map engine to handle the rendering and interaction of map tiles and vector features.

[0045] The backend business logic layer is based on the Node.js runtime environment. It leverages its event-driven and non-blocking I / O features to handle high-concurrency file streams; it integrates the GDAL (Geospatial Data Abstraction Library) engine, and implements high-performance geographic data parsing, coordinate transformation, and tiling processing on the server side through the `gdal-async` binding library.

[0046] Specifically, the backend is based on a Node.js environment, and the gdal-async library is installed via the NPM package manager. This library provides asynchronous Node.js bindings for the GDAL C++ library. In the code, by importing this module using `require('gdal-async')`, you can directly call its asynchronous methods such as `gdal.open()`, `dataset.bounds()`, and `coordinateTransformation()`, achieving high-performance, non-blocking read / write and coordinate calculations for dozens of GIS formats, including GeoTIFF and Shapefile.

[0047] The data storage layer employs a "file system + metadata sidecar" storage strategy. It abandons the traditional heavy-duty relational database approach to storing metadata, instead storing metadata directly in the file system in JSON format.

[0048] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for managing lightweight multi-source GIS data based on a Sidecar mode, characterized in that: Includes the following steps: S1. Receive the GIS data file uploaded by the user, and parse the spatial information and attribute information of the GIS data file through the backend GDAL engine; S2. Generate a metadata file corresponding to the GIS data file based on the Sidecar pattern, and store the metadata file and the GIS data file in the same directory; S3. When it is necessary to display the GIS data on a Web map, the spatial range of the data is transformed according to the coordinate projection information in the metadata file to unify it into a coordinate system compatible with the Web map. S4. When a preview request for vector format GIS data is received, the corresponding binary vector file is parsed in real time, dynamically converted into a format that can be rendered by the front end, and returned. S5. Monitor the system storage status and perform space prediction before uploading.

2. The lightweight multi-source GIS data management method based on the Sidecar mode according to claim 1, characterized in that: Step S2 includes the following sub-steps: S21. Based on the parsing results, create a metadata file in .meta.json format with the same name as the GIS data file in the same directory; S22. Write the original name, size, modification time, spatial range, and coordinate projection information of the file into the metadata file; S23. In response to the file list request, read multiple .meta.json files in the directory and assemble them into a list and return it.

3. The lightweight multi-source GIS data management method based on the Sidecar mode according to claim 1, characterized in that: Step S3 includes the following sub-steps: S31. Read coordinate projection information from the metadata file; S32. If the projection information is a standard EPSG code, then it shall be used directly; S33. If the projection information is a custom projection definition, the spatial range of the data is forcibly converted from the source coordinate system to the WGS84 coordinate system through the GDAL engine.

4. The lightweight multi-source GIS data management method based on the Sidecar mode according to claim 1, characterized in that: Step S4 includes the following sub-steps: S41. Verify the integrity of the requested vector data file; S42. Real-time reading of the geometric and attribute binary streams of the vector data; S43. Dynamically parse and concatenate the data into a GeoJSON format data stream in memory; S44. Return the data stream to the front end for rendering.

5. The lightweight multi-source GIS data management method based on the Sidecar mode according to claim 1, characterized in that: Step S5 includes the following sub-steps: S51. Identify the operating system and call the corresponding disk space query interface; S52. Calculate the current disk usage. S53. Based on the size of the file to be uploaded and the remaining space, make a prediction. If there is insufficient space, stop the upload and issue an alarm.

6. The lightweight multi-source GIS data management method based on the Sidecar mode according to claim 3, characterized in that, In step S33, the forced conversion is completed on the server side, and the conversion result is only the spatial range coordinates for front-end positioning, without modifying the original data file.

7. The lightweight multi-source GIS data management method based on the Sidecar mode according to claim 4, characterized in that: In step S43, the dynamically generated GeoJSON format data is only for this preview and will not be persistently stored. 8.A lightweight multi-source GIS data management system based on a Sidecar mode, characterized in that, The system adopts a front-end / back-end separation B / S architecture and includes: The front-end presentation layer is used for map rendering, user interaction, and data requests. The backend business logic layer is used to execute steps S1 to S5 as described in claim 1. The data storage layer is used to store GIS data files and their corresponding Sidecar metadata files.

Images