A compact storage and management method and system for real scene three-dimensional data products
By constructing a database and using ETL technology, the problem of diverse and scattered real-scene 3D data formats was solved, achieving unified management and accurate sharing of data, and ensuring data consistency and traceability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 湖南省第一测绘院
- Filing Date
- 2025-09-04
- Publication Date
- 2026-05-12
AI Technical Summary
Real-world 3D data comes in various formats, is scattered and inconsistent, making it difficult to manage and share effectively, and lacks mechanisms for verifying data accuracy and tracing its origin.
By constructing an original database, establishing standards for real-scene 3D data products, utilizing ETL technology for data extraction, transformation, and loading, and combining data mapping and reconstruction, metadata tables are generated and data verification is performed to ensure data consistency and integrity.
It enables efficient and compact storage and management of real-scene 3D data, ensuring the accuracy and traceability of data sharing, and supporting high-quality data exchange and management.
Smart Images
Figure CN121071174B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer information technology, specifically to a compact storage and management method and system for real-scene 3D data products. Background Technology
[0002] Currently, the types of real-scene 3D data elements include vector data, raster data, 3D model data, video data, image data, and relational data. These data are stored in different data formats and scattered in different locations in the file system directory. The data organization methods are inconsistent, the data description information is incomplete, and it is not convenient to manage, display, or share data. Furthermore, the distribution mechanism is simple, data copies can be copied repeatedly, data content can be modified manually, different departments have different standards for expressing the same geographic entity, and the accuracy of the data is not verified, making it impossible to ensure the accuracy of shared data and trace the data production unit. Summary of the Invention
[0003] To address the problems of inconvenient data storage solutions for real-scene 3D platforms for viewing in desktop GIS software, diverse real-scene 3D data formats hindering unified management, and lack of traceability for shared data, this invention proposes a compact storage and management method for real-scene 3D data products. The method includes:
[0004] Acquire real-world 3D data and construct an original database. The real-world 3D data includes geographic entity data, geographic scene data, IoT sensing data, and map data.
[0005] Create a task to assemble a real-scene 3D data product and formulate content standards for the real-scene 3D data product.
[0006] The content standards of the real-scene 3D data products are compared with the real-scene 3D database standards to obtain the conversion relationship of the data content. A geographic entity data mapping table is compiled based on the conversion relationship. Based on the geographic entity data mapping table, ETL technology is used to extract, transform, and load the original database data to reconstruct the structure of the geographic entity data. For data at the boundary, in order to ensure data integrity, the data is fully exported using intersection rules without pruning or filtering, and the geographic entity data metadata is recorded to realize the assembly of geographic entity data content.
[0007] According to the data range, data format, and specified center point coordinates and tile segmentation requirements stipulated in the assembly task, the geographic scene data in 3dtile format in the original database is converted into metadata table, organizational structure data table and tile file data table and stored in the real scene 3D data product file. The organizational structure data table and the tile file data table are associated through an index, and the assembled geographic scene data items are added to the real scene 3D data product data directory.
[0008] Based on the assembly task, the content and scope of IoT sensing data are selected, data is extracted from the original database, and for IoT sensing data that requires changes to its data structure, a data mapping table is established. The IoT sensing data is then converted using an ETL data conversion tool, and the assembled IoT sensing data items are added to the real-scene 3D data product data catalog.
[0009] Based on the entity and scene data list of the map data, the map configuration information is readjusted, and layers whose data sources in the map have not been assembled are identified as invalid layers. Invalid layers are then deleted using an invalid layer cleanup tool to assemble the map data content.
[0010] The assembled geographic entity data, geographic scene data, IoT sensing data, and map data are written into the real-scene 3D data product. The verification value of the data in the real-scene 3D database is calculated and compared with the verification value of the real-scene 3D data product. The consistency of the verification values is compared to complete the data verification.
[0011] Save all configuration and verification information during the data product assembly process.
[0012] Furthermore, the assembly task includes product name, data range, and data time point, and the real-scene 3D data product content standard includes geographic entity identification data structure definition, modal data list and data structure definition, IoT sensing data list and data structure definition, relational data list and data structure definition, geographic scene data list and data format, symbol data list and data structure definition, and map data list and data format.
[0013] Furthermore, the geographic entity data includes geographic entity identifier data, modal data, relational data, and symbolic data, with the symbolic data stored in a data file in JSON format.
[0014] Furthermore, the regeneration of geographic scene data includes: when the geographic scene data center point in the real-world 3D data product is inconsistent with the geographic scene data center point in the platform, the center point of all data needs to be adjusted, and the offset distance of the center point cannot exceed the validity that a floating-point type can express; when the tile assembly requirements are inconsistent with the data tile partitioning scheme in the system, the tile data needs to be re-partitioned.
[0015] Furthermore, the IoT sensing data is divided into three categories: structured data, semi-structured data, and unstructured data. The semi-structured and unstructured data are saved to the data product file using BLOB fields according to data type, and a unified spatiotemporal identifier is added to form a structured relational data table.
[0016] Furthermore, the map data includes map configuration information and map service information.
[0017] Furthermore, the MD5 algorithm is used to calculate the verification value of the data in the real-scene 3D database. During the verification process, the data is verified in batches according to the size of the data.
[0018] This application also provides a compact storage and management system for real-scene 3D data products, used to implement the method described in the above claims, the system comprising:
[0019] The data acquisition and data product assembly module is used to acquire real-scene 3D data, create real-scene 3D data product assembly tasks, and formulate real-scene 3D data product content standards. The real-scene 3D data includes geographic entity data, geographic scene data, IoT sensing data, and map data. The assembly task includes product name, data range, and data time point. The real-scene 3D data product content standards include geographic entity identifier data structure definition, modal data list and data structure definition, IoT sensing data list and data structure definition, relational data list and data structure definition, geographic scene data list and data format, symbol data list and data structure definition, and map data list and data format.
[0020] The geographic entity data assembly module is used to compare the content standard of the real-scene 3D data product with the real-scene 3D database standard, compile a geographic entity data mapping table, clarify the conversion relationship of data content, and use ETL technology to extract, transform and load the original database data to reconstruct the structure of geographic entity data based on the geographic entity data mapping table. For data at the boundary, in order to ensure data integrity, the data is fully exported using intersection rules without pruning or filtering, and the geographic entity data metadata is recorded to realize the assembly of geographic entity data content.
[0021] The geographic scene data assembly module is used to regenerate geographic scene data according to the data range, data format, and specified center point coordinates and tile segmentation requirements stipulated in the assembly task, thereby assembling geographic scene data content and recording geographic scene metadata.
[0022] The IoT sensing data assembly module is used to establish an IoT sensing data product mapping table based on the IoT sensing data type, data range, and specified data format selected in the assembly task, thereby assembling IoT sensing data content and recording IoT sensing metadata.
[0023] The map data assembly module is used to readjust the map configuration information according to the map data list to assemble the map data content.
[0024] The data product generation and verification module is used to write the assembled geographic entity data, geographic scene data, IoT sensing data, and map data into the real-scene 3D data product, calculate the verification value of the data in the real-scene 3D database and the verification value of the real-scene 3D data product, compare the consistency of the verification values, and complete the data verification; after verification, it saves all configuration information and verification information of the data product assembly process.
[0025] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it achieves compact storage and management of real-scene 3D data products based on the method described above.
[0026] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, enables compact storage and management of real-scene 3D data products based on the method described above.
[0027] The compact storage and management method for real-scene 3D data products provided in this application utilizes real-scene 3D data production specifications and combines functions such as data mapping, data structure reconstruction, and data verification. It filters, cleans, transforms, and assembles real-scene 3D platform data to generate real-scene 3D exchange data packets. Then, through data verification, it ensures that the data packets are consistent with the real-scene 3D platform data. At the same time, the verification information of the data packets is recorded on the real-scene 3D platform, ultimately achieving efficient and high-quality sharing and management of real-scene 3D data. Attached Figure Description
[0028] Figure 1 A flowchart illustrating a compact storage and management method for real-world 3D data products.
[0029] Figure 2 A block diagram of a compact storage and management system for real-world 3D data products. Detailed Implementation
[0030] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.
[0031] As one implementation scheme, Figure 1 This is a flowchart illustrating a compact storage and management method for real-scene 3D data products, as described in the embodiments of this application. Figure 1 As shown, the method includes:
[0032] Step 1: Acquire real-scene 3D data and construct the original database. The real-scene 3D data includes geographic entity data, geographic scene data, IoT sensing data, and map data.
[0033] Step 2: Create a task to assemble a real-scene 3D data product and formulate content standards for the real-scene 3D data product.
[0034] Before assembling the data, the detailed content of the real-scene 3D data product assembly needs to be determined. The assembly task includes, but is not limited to, information such as product name, product standard description, product supplementary description, data range, and data time point. Specifically, the task name is used to generate the filename of the data package; the data range is used to determine the location for data export, which can be an imported range line or an administrative division surface; the data time point is used to determine the time information of the packaged data, supporting full-range export of a version at a specified time point or incremental data export over a time period; it is also necessary to specify which types of data need to be packaged, and the packaging information is saved in the data package in the form of a relational data table.
[0035] Product content standards can be customized according to requirements. The real-scene 3D data product content standards include, but are not limited to, definitions of geographic entity identification data structures, modal data lists and data structures, IoT sensing data lists and data structures, relational data lists and data structures, geographic scene data lists and data formats, symbol data lists and data structures, and map data lists and data formats. Modal data refers to the inherent form used to describe geographic entities that can be perceived and identified by humans or automated devices. Depending on the method of expressing geographic entity characteristics, modal types are divided into vector graphics, 3D models, raster data, point cloud data, crowdsourced images, video data, parametric models, and other data. Relational data specifically refers to spatial relationships, attribute relationships, and temporal relationships between entities. Entity relationships are recorded and expressed using relation tables, triples, and other forms.
[0036] Step 3: Compile a geographic entity data mapping table to assemble geographic entity data.
[0037] Geographic entity identification data, modal data, and relational data are typically stored in a database as relational data tables. By comparing the standards of real-world 3D databases and data product content standards, a geographic entity data mapping table is created to clarify the conversion relationships of the data content. This data mapping table includes, but is not limited to, table mapping, field mapping, encoding mapping, and attribute item mapping. Predefined geographic entity data mapping table templates can also be selected. Modal data generally has designated fields for storing the modal data body; for example, the field name for geometry is usually [geometry]. By utilizing the row and column organization characteristics of relational data tables, the data mapping table can be created to complete the conversion between different data standards. Field mapping relationships require four unique constraints: database name, table name, field name, and filter conditions. Predefined general geographic entity data mapping tables can also be selected.
[0038] Based on the geographic entity data mapping table, ETL technology is used to extract, transform, and load the original database data to complete the assembly of geographic entity data. Vector data geometric fields use WKT format; the principle for edge data processing is as follows: for data located at boundaries, to ensure data integrity, all data is exported using intersection rules without pruning or filtering. Symbolic data is stored in data files in JSON format, which includes computer- and human-readable self-descriptive information for the symbols and establishes unique identifiers for the symbolic data.
[0039] Step 4: Geographic Scene Data Assembly. Based on the data range, data format, specified center point coordinates, and tile partitioning requirements defined in the assembly task, the geographic scene data in 3dtile format from the original database is converted into a metadata table, an organizational structure data table, and a tile file data table, which are then stored in the reality 3D data product file. The organizational structure data table and the tile file data table are linked through an index. The assembled geographic scene data items are then added to the reality 3D data product data directory.
[0040] For tile data that exceeds the data range, all tiles must be included. When the center point of the data product is inconsistent with the center point of the data center in the platform, the center point of all data needs to be adjusted, and the offset distance of the center point cannot exceed the validity of floating-point type. When the tile assembly and segmentation requirements are inconsistent with the data tile segmentation scheme in the system, the tile data needs to be re-segmented. For scene data of image types such as DEM and DOM, the data is segmented, cached hierarchically, a hierarchical segmentation index is established, and stored in the data product file.
[0041] Step 5: IoT Sensing Data Assembly. Based on the assembly task, select the content and scope of the IoT sensing data, extract data from the original database, and for IoT sensing data that requires data structure modification, establish a data mapping table. Use an ETL data conversion tool to transform the IoT sensing data and add the assembled IoT sensing data items to the real-scene 3D data product data catalog.
[0042] In a real-world 3D system, IoT sensing data possesses a unified spatiotemporal identifier and shares a unified spatiotemporal reference with geographic entity data, establishing a correlation through spatiotemporal and attribute information. IoT sensing data is categorized into three types based on its data structure: structured data, such as environmental monitoring data, high-precision positioning data, and business data; semi-structured data, such as JSON and XML formats; and unstructured data, such as images, audio / video streams, and internet data. Semi-structured / unstructured IoT sensing data is stored in data product files using BLOB fields according to data type, and a unified spatiotemporal identifier is added to form a structured relational data table. The IoT sensing data product mapping table compilation scheme references the geographic entity data mapping table.
[0043] Step 6: Map Data Construction. Based on the entity and scene data list of the map data, readjust the map configuration information. Determine invalid layers whose data sources have not been assembled as invalid layers, and delete them using the invalid layer cleanup tool to assemble the map data content.
[0044] The map includes 2D and 3D maps, with 3D maps sometimes referred to as scenes; map data includes map configuration information and map services; during map data construction, the map configuration information needs to be readjusted based on the exported data list, which includes, but is not limited to, layer directory, layer attributes, layer order, layer symbols, layer display level, map coordinate system, and display coordinate system; map services include, but are not limited to, WMS, WMTS, MVT, etc.; only user-defined maps and system maps can be assembled, and map data from unauthorized users cannot be downloaded.
[0045] Step 7: Data Product Generation and Verification. The assembled geographic entity data, geographic scene data, IoT sensing data, and map data are written into the real-scene 3D data product. The verification values of the data in the real-scene 3D database are calculated and compared with the verification values of the real-scene 3D data product. The consistency of the verification values is then verified to complete the data verification.
[0046] The coordinate system information involved in the data needs to be added to the data product. The coordinate information should conform to the OpenGIS® 01-009 standard. The value range information of the fields involved in the data also needs to be added to the data product file. The checksum is calculated using algorithms including but not limited to MD5. During the checksum process, the data can be checked in batches according to the data volume. The data product uses a data format based on the SQLite extension and is stored as a single data file with the data file extension "rs3db".
[0047] Step 8: Save the data product assembly record. Save all configuration and verification information from the data product assembly process. This facilitates data traceability and can also be used to verify incremental packages during incremental updates.
[0048] The method proposed in this embodiment utilizes the production specifications for real-scene 3D data, combined with functions such as data mapping, data structure reconstruction, and data verification. It filters, cleans, transforms, and assembles real-scene 3D platform data to generate real-scene 3D exchange data packets. Then, through data verification, it ensures that the data packets are consistent with the real-scene 3D platform data. At the same time, the verification information of the data packets is recorded on the real-scene 3D platform, ultimately achieving efficient and high-quality sharing and management of real-scene 3D data.
[0049] As an alternative implementation scheme Figure 2 This is a block diagram of a compact storage and management system for real-scene 3D data products, as described in the embodiments of this application. Figure 2 As shown, the system includes:
[0050] The data acquisition and data product assembly module is used to acquire real-scene 3D data, create real-scene 3D data product assembly tasks, and formulate real-scene 3D data product content standards. The real-scene 3D data includes geographic entity data, geographic scene data, IoT sensing data, and map data. The assembly task includes product name, data range, and data time point. The real-scene 3D data product content standards include geographic entity identifier data structure definition, modal data list and data structure definition, IoT sensing data list and data structure definition, relational data list and data structure definition, geographic scene data list and data format, symbol data list and data structure definition, and map data list and data format.
[0051] The geographic entity data assembly module is used to compare the content standard of the real-scene 3D data product with the real-scene 3D database standard, compile a geographic entity data mapping table, clarify the conversion relationship of data content, and use ETL technology to extract, transform and load the original database data to reconstruct the structure of geographic entity data based on the geographic entity data mapping table. For data at the boundary, in order to ensure data integrity, the data is fully exported using intersection rules without pruning or filtering, and the geographic entity data metadata is recorded to realize the assembly of geographic entity data content.
[0052] The geographic scene data assembly module is used to regenerate geographic scene data according to the data range, data format, and specified center point coordinates and tile segmentation requirements stipulated in the assembly task, thereby assembling geographic scene data content and recording geographic scene metadata.
[0053] The IoT sensing data assembly module is used to establish an IoT sensing data product mapping table based on the IoT sensing data type, data range, and specified data format selected in the assembly task, thereby assembling IoT sensing data content and recording IoT sensing metadata.
[0054] The map data assembly module is used to readjust the map configuration information according to the map data list to assemble the map data content.
[0055] The data product generation and verification module is used to write the assembled geographic entity data, geographic scene data, IoT sensing data, and map data into the real-scene 3D data product, calculate the verification value of the data in the real-scene 3D database and the verification value of the real-scene 3D data product, compare the consistency of the verification values, and complete the data verification; after verification, it saves all configuration information and verification information of the data product assembly process.
Claims
1. A compact storage and management method for real-scene 3D data products, characterized in that, The method includes: Acquire real-world 3D data and construct an original database. The real-world 3D data includes geographic entity data, geographic scene data, IoT sensing data, and map data. Create a task to assemble a real-scene 3D data product and formulate content standards for the real-scene 3D data product; The content standards of the real-scene 3D data products are compared with the real-scene 3D database standards to obtain the conversion relationship of the data content. A geographic entity data mapping table is compiled based on the conversion relationship. Based on the geographic entity data mapping table, ETL technology is used to extract, transform and load the original database data to reconstruct the structure of the geographic entity data. For data at the boundary, in order to ensure data integrity, the data is fully exported using intersection rules without pruning or filtering. The geographic entity data metadata is recorded to realize the assembly of geographic entity data content. According to the data range, data format, and specified center point coordinates and tile segmentation requirements stipulated in the assembly task, the geographic scene data in 3dtile format in the original database is converted into metadata table, organizational structure data table and tile file data table and stored in the real scene 3D data product file. The organizational structure data table and the tile file data table are associated through an index, and the assembled geographic scene data items are added to the real scene 3D data product data directory. Based on the assembly task, select the content and scope of IoT sensing data, extract data from the original database, establish a data mapping table for IoT sensing data that needs to change its data structure, use ETL data conversion tools to convert IoT sensing data, and add the assembled IoT sensing data items to the real scene 3D data product data catalog. Based on the entity and scene data list of the map data, the map configuration information is readjusted, and the layers whose data sources in the map have not been assembled are identified as invalid layers. Invalid layers are deleted using the invalid layer cleanup tool to achieve map data content assembly. The assembled geographic entity data, geographic scene data, IoT sensing data, and map data are written into the real-scene 3D data product. The verification value of the data in the real-scene 3D database is calculated and compared with the verification value of the real-scene 3D data product. The consistency of the verification values is compared to complete the data verification. Save all configuration and verification information during the data product assembly process; The real-scene 3D data product assembly task includes product name, data range, and data time point. The real-scene 3D data product content standard includes geographic entity identification data structure definition, modal data list and data structure definition, IoT sensing data list and data structure definition, relational data list and data structure definition, geographic scene data list and data format, symbol data list and data structure definition, and map data list and data format.
2. The method according to claim 1, characterized in that, The geographic entity data includes geographic entity identifier data, modal data, relational data, and symbolic data. The symbolic data is stored in a data file in JSON format.
3. The method according to claim 1, characterized in that, The process of regenerating geographic scene data includes: when the geographic scene data center point in the real-world 3D data product is inconsistent with the geographic scene data center point in the real-world 3D database, it is necessary to adjust the center point of the geographic scene data within a specified range in the real-world 3D data product, and the offset distance of the center point cannot exceed the validity that a floating-point type can express; when the tile assembly requirements are inconsistent with the data tile partitioning scheme in the system, it is necessary to readjust the tile data partitioning size and adjust the data center point.
4. The method according to claim 1, characterized in that, The IoT sensing data is divided into three categories: structured data, semi-structured data, and unstructured data. The semi-structured and unstructured data are saved to the data product file using BLOB fields according to data type, and a unified spatiotemporal identifier is added to form a structured relational data table.
5. The method according to claim 1, characterized in that, The map data includes map configuration information and map service information.
6. The method according to claim 1, characterized in that, The MD5 algorithm is used to calculate the verification value of the data in the real-scene 3D database. During the verification process, the data is verified in batches according to the size of the data.
7. A compact storage and management system for real-scene 3D data products, used to implement the method described in any one of claims 1-6, characterized in that, The system includes: a data acquisition and data product assembly module, used to acquire real-scene 3D data, create real-scene 3D data product assembly tasks, and formulate real-scene 3D data product content standards. The real-scene 3D data includes geographic entity data, geographic scene data, IoT sensing data, and map data. The assembly task includes product name, data range, and data time point. The real-scene 3D data product content standards include geographic entity identifier data structure definition, modal data list and data structure definition, IoT sensing data list and data structure definition, relational data list and data structure definition, geographic scene data list and data format, symbol data list and data structure definition, and map data list and data format. A geographic entity data assembly module is used to compare the real-scene 3D data product content standards with the real-scene 3D database standards, compile a geographic entity data mapping table, clarify the conversion relationships of data content, and, based on the geographic entity data mapping table, use ETL technology to extract, transform, and load the original database data to reconstruct the geographic entity data structure. For data at boundaries, intersection rules are used to ensure data integrity. The system performs the following steps: Full export, no cropping or filtering, recording geographic entity data metadata to assemble geographic entity data content; Geographic scene data assembly module, used to regenerate geographic scene data according to the data range, data format, specified center point coordinates, and tile segmentation requirements defined in the assembly task, assembling geographic scene data content and recording geographic scene metadata; IoT sensing data assembly module, used to establish an IoT sensing data product mapping table according to the IoT sensing data type, data range, and specified data format selected in the assembly task, assembling IoT sensing data content and recording IoT sensing metadata; Map data assembly module, used to readjust map configuration information according to the map data list, assembling map data content; Data product generation and verification module, used to write the assembled geographic entity data, geographic scene data, IoT sensing data, and map data into the real-scene 3D data product, calculate the verification value of the data in the real-scene 3D database and the verification value of the real-scene 3D data product, compare the consistency of the verification values, and complete data verification; After verification, all configuration information and verification information of the data product assembly process are saved.
8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it realizes compact storage and management of real-scene 3D data products based on the method described in any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, wherein when executed by a processor, the computer program enables compact storage and management of real-scene 3D data products based on the method described in any one of claims 1-6.