A spatial perception data storage method and system for multi-source data fusion

Abstract

The present invention discloses a spatial perception data storage method and system for multi-source data fusion, and relates to the technical field of three-dimensional spatial data storage. The present invention significantly improves the integrity and accuracy of data by respectively collecting point cloud data, panoramic data and IoT perception data and performing preprocessing operations respectively; multi-source data fusion and lightweight processing realize the efficient fusion of point cloud data, panoramic data and IoT perception data, and effectively reduces the cost of data storage and transmission; combining point cloud data, panoramic images and IoT perception data for data fusion spatial database storage can integrate multiple data sources, provide comprehensive spatial perception information, and realize efficient spatial perception data storage; based on the multi-source data spatial query method and the real-time data interaction between the spatial database and the Web end, the real-time real-scene three-dimensional rendering of the spatial perception data model is realized, and the interactivity and real-time performance of data application are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional spatial data storage, and in particular to a spatial perception data storage method and system for multi-source data fusion. Background Art

[0002] Spatial perception technology refers to the technology that obtains information about the surrounding environment through various sensors, and then perceives information such as the spatial position, orientation and distance around itself. It is the basis for 3D modeling and virtual reality technology to achieve spatial data acquisition. 3D modeling technology uses the data collected by spatial perception technology to create a digital 3D model of the physical world. Virtual reality (VR) technology uses the models created by 3D modeling technology to provide users with an immersive visual and sensory experience. Interactions in VR environments often require highly accurate spatial perception to simulate the dynamics of the real world.

[0003] With the development of 3D modeling and virtual reality technology, the demand for accurate spatial perception data storage is growing. Existing data storage methods often rely on a single data source, which can only provide spatial information of a certain dimension and cannot fully reflect complex spatial structures and environments. For example, point cloud data can accurately describe the 3D shape of an object, but lacks color and texture information; panoramic image data can provide all-round visual information, but lacks depth information; IoT perception data can monitor environmental parameters, but cannot directly provide spatial structure information.

[0004] In practical applications, it is often necessary to integrate multiple data sources to provide comprehensive and accurate spatial perception information. However, the data formats, precisions, and collection methods of different data sources are different, which makes data fusion difficult. Existing technologies lack effective methods and tools for data fusion, resulting in low data utilization and unsatisfactory fusion effects. In addition, the collection, preprocessing, and storage processes of multi-source data are complex, involving a large amount of data processing and computing operations, and have high requirements for processing algorithms and hardware performance. Due to the large amount of data and complex processing, existing systems are difficult to complete data rendering in a short time and cannot meet the needs of real-time applications. Summary of the invention

[0005] In view of the limitations of existing spatial perception data storage, the present invention proposes a spatial perception data storage method and system with multi-source data fusion, which comprehensively captures and stores spatial information by fusing point cloud data, panoramic data and IoT perception data, and realizes efficient storage and retrieval of data.

[0006] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0007] A spatially aware data storage method for multi-source data fusion, the method comprising:

[0008] Collect point cloud data, panoramic data and IoT perception data through multiple devices or sensors;

[0009] Preprocessing the point cloud data, panoramic data and IoT sensing data respectively;

[0010] The pre-processed point cloud data, panoramic data and IoT perception data are fused into spatial perception data, and the spatial perception data is lightweight processed to generate a unified spatial perception data model;

[0011] Use variable node octree algorithm to construct hierarchical details and slice data of point cloud data;

[0012] Encoding the spatial identity of the point cloud data and the panoramic data, establishing a spatiotemporal index, and storing the spatial perception data model in a spatial database in the same geographic coordinate system;

[0013] Based on a multi-source data spatial query method and real-time data interaction between a spatial database and a Web terminal, the spatial perception data model is subjected to real-time three-dimensional rendering.

[0014] As a preferred solution of the present invention, the point cloud data is preprocessed, including:

[0015] Time synchronization: Use GPS timestamps or system timestamps to synchronize point cloud data from different devices, and use linear interpolation to estimate missing data points at the specified timestamp. The linear interpolation formula is:

[0016] ;

[0017] In the formula, and All timestamps are known , The point cloud coordinates under, t is the target timestamp, is the point cloud coordinate at the target timestamp;

[0018] Scale analysis: For point cloud data from different devices or acquired under different conditions, select objects of known size or use the calibration information provided by the device to estimate the scale;

[0019] Determine the average scale or median scale of all point cloud data as the standard scale;

[0020] Calculating a scaling factor of each point cloud relative to a standard scale, and applying the scaling factor to adjust the scale of the point cloud;

[0021] Spatial alignment: Select a reference point cloud as the benchmark, use the iterative closest point algorithm to rotate and translate the other point cloud data outside the reference point cloud, and minimize the objective function ,in and are the matching point pairs, n is the matching point pair index; R is the rotation matrix, T is the translation vector; N is the number of matching point pairs;

[0022] Point cloud denoising: Apply statistical filters to remove isolated points in the point cloud data based on the average distance of each point's neighbors;

[0023] Point cloud filtering: Smoothing point cloud data by moving least squares method;

[0024] Distortion correction: A fast polynomial distortion model is used to remove distortion in point cloud data. Specifically:

[0025] Before system deployment, use a calibration plate or known structure to obtain the distortion coefficient, integrate the correction formula into the data stream, and use multi-core processors for parallel calculation. The correction formula is:

[0026] ;

[0027] In the formula, x and y are the original coordinates, , is the coordinate after distortion correction; r is the distance from the point to the center of the image, k1, k2, k3 are all radial distortion coefficients;

[0028] For devices with known distortion patterns, a lookup table (LUT) is pre-calculated and fast correction is performed by looking up the table in real-time applications.

[0029] As a preferred solution of the present invention, the panoramic data is preprocessed, including:

[0030] Image denoising: Apply bilateral filter or median filter to remove image noise;

[0031] Image correction: Use camera correction parameters to adjust the position of each pixel according to the radial and tangential distortion coefficients to eliminate optical distortion;

[0032] Resolution matching: Use bicubic interpolation for image scaling and adjust the image resolution to ensure that it matches the point cloud data. The formula is:

[0033] ;

[0034] In the formula, is the original image; , b is the original image The index in is used to traverse all pixels in the original image; Original image Index , b related pixel coordinates; is the resampled image, , is the new pixel coordinate in the resampled image; h is the interpolation kernel function;

[0035] Image stitching: Use the SIFT algorithm to detect feature points and the RANSAC algorithm to estimate the transformation and stitch the images into panoramic data;

[0036] Color consistency adjustment: Apply color balancing technology based on histogram matching to perform color balancing on the stitched panoramic data to eliminate image stitching gaps;

[0037] Image enhancement: Apply the limited contrast adaptive histogram equalization algorithm to perform local contrast enhancement.

[0038] As a preferred solution of the present invention, the IoT sensing data is preprocessed, including:

[0039] Sensor data parsing: Parsing raw sensor data into a structured format;

[0040] Time synchronization: Timestamp each piece of IoT sensing data. If the data comes from different sensors and the timestamps are different, they are corrected according to the server time or GPS time, and synchronized by interpolation or alignment to the nearest integer second.

[0041] Normalization: Normalize the IoT sensing data and scale all data to the [0,1] range;

[0042] Data association: According to timestamp and location information, IoT sensing data is associated with objects in point cloud data and panoramic data to establish a mapping relationship between multi-source data.

[0043] As a preferred solution of the present invention, the steps of implementing the fusion processing and lightweight processing include:

[0044] Use spatial reference coordinate conversion and time difference technology to perform time synchronization and spatial alignment on the pre-processed point cloud data, panoramic data and IoT perception data;

[0045] The farthest point sampling method with improved curvature is used to perform semantic segmentation on point cloud data, and the improved Go-ICP algorithm is used to fit the point cloud structure in multiple time and space.

[0046] Create a unified set of semantic labels across all data sources, covering all entities and attributes;

[0047] Use NLP technology to identify entities in text, parse synonyms of entities, and build a mapping table to map the identified and parsed entities into a unified semantic framework;

[0048] Leverage the pre-trained BERT model to understand and map semantic labels and attributes;

[0049] Input the spatiotemporally aligned and semantically matched point cloud data, panoramic data, and IoT sensing data into the multidimensional data integration model, and apply the Bayesian network to infer possible missing data or resolve data conflicts;

[0050] Generative adversarial networks and variational autoencoder technologies are used to fit point clouds and complete incomplete panoramic images at multiple time and space angles, and to perform probability distribution modeling for the missing parts;

[0051] Generate standardized spatial perception data with spatiotemporal fusion by using the same semantic mapping method;

[0052] The features of spatial perception data are extracted through principal component analysis or autoencoder, and the spatial perception data is optimized and compressed through differential pulse code modulation technology, encoded into a lightweight spatial perception data model, and imported into the spatial database.

[0053] As a preferred solution of the present invention, the improved Go-ICP algorithm comprises:

[0054] Nested BnB search: decomposes the search for rotation and translation into two independent levels, first searching for rotation parameters, and then searching for translation parameters based on the determined rotation parameters;

[0055] Registration error: The registration error is calculated by comparing the distances between the corresponding closest points in the source point cloud and the target point cloud;

[0056] Initial cube setting: Set the minimum search space cube and set the initial estimated pose as the side length;

[0057] Uncertainty radius definition: Define an uncertainty radius during the search process to limit the search space;

[0058] Iterative optimization: Use iterative optimization pose to obtain the closest point pair between the source point cloud and the target point cloud, and update the pose of the source point cloud. The formula is:

[0059] ;

[0060] In the formula, It’s a new posture. It's the old position. is the transformation matrix calculated based on the nearest point pair information; K is the pose optimization coefficient;

[0061] Weight update: Update the point pair weights according to the distance and angle difference between the point pairs. The formula is:

[0062] cos( );

[0063] In the formula, is the weight of the point pair (i, j), d is the Euclidean distance between the point pairs (i, j), is the angle difference between the point pair (i, j), and is the iterative optimization coefficient;

[0064] Convergence judgment: Set the registration error threshold and iteration limit. When the registration error during the iteration process is less than the preset registration error threshold or the number of iterations reaches the iteration limit, the iteration is stopped and the registration process is considered to have converged.

[0065] As a preferred solution of the present invention, the method of using a variable node octree algorithm to construct hierarchical details and slice data on point cloud data includes:

[0066] Determine the root node boundary of the octree according to the minimum and maximum coordinate values ​​of the point cloud data;

[0067] Recursively assign each point in the point cloud data to an octree node until the preset tree depth is reached or the number of points in the node reaches an upper limit;

[0068] Each octree node represents a data slice, and the detail level of the data slice is automatically adjusted according to the user's perspective, specifically: capturing the user's zoom operation or view change event, and selecting the octree depth according to the user's zoom level or view distance; analyzing the user's current viewpoint position and line of sight direction in real time, selecting the octree node aligned with the user's line of sight direction, and preferentially loading the point cloud data in the octree node.

[0069] As a preferred solution of the present invention, the point cloud data and the panoramic data are spatially encoded, and a spatiotemporal index is established, and the spatial perception data model is stored in a spatial database in the same geographic coordinate system, specifically including:

[0070] Use Geohash or S2 geometry libraries to assign unique spatial identity codes to point cloud data and panoramic data;

[0071] A composite spatiotemporal index is constructed by combining the timestamp and the spatial coding, the spatially aware data model is stored in a spatial database, and the storage and access of data are optimized through the spatiotemporal index.

[0072] As a preferred solution of the present invention, real-time three-dimensional rendering of the spatial perception data model specifically includes:

[0073] Get the target to be rendered according to the current view frustum coordinates, and at the same time get the view frustum space geometry information of the previous rendering scene;

[0074] Calculate the newly added field of view range information based on the current view frustum coordinates and the view frustum space geometry information of the previous scene;

[0075] Send a data call request containing the newly added field of view information to the server;

[0076] After receiving the data call request, the server performs geometric interleaving query on the point cloud data in the spatial database based on the newly added field of view information;

[0077] After the query is completed, the server sends the newly added spatiotemporal index to the client;

[0078] The client queries the corresponding point cloud data according to the newly added spatiotemporal index, renders the target, and maps the corresponding panoramic data according to the spatiotemporal index of the queried point cloud data, and preloads the panoramic data.

[0079] A spatial perception data storage system for multi-source data fusion, based on the spatial perception data storage method for multi-source data fusion as described above, the system comprises:

[0080] The data acquisition module is used to collect point cloud data, panoramic data and IoT perception data through a variety of devices or sensors;

[0081] A data preprocessing module, used to perform preprocessing operations on the point cloud data, panoramic data and IoT sensing data respectively;

[0082] A data fusion module is used to fuse the pre-processed point cloud data, panoramic data and IoT perception data into spatial perception data, and to perform lightweight processing on the spatial perception data to generate a unified spatial perception data model;

[0083] Octree module, used to construct hierarchical details and slice data of point cloud data using variable node octree algorithm;

[0084] An index encoding module, used to perform spatial identity encoding on point cloud data and panoramic data, and to establish a spatiotemporal index, and to store the spatial perception data model in a spatial database in the same geographic coordinate system;

[0085] The real scene rendering module is used to perform real scene three-dimensional rendering of the spatial perception data model in real time based on a multi-source data spatial query method and real-time data interaction between a spatial database and a Web terminal.

[0086] The beneficial effects of the present invention are as follows: point cloud data, panoramic data and IoT perception data are collected respectively by a variety of sensor devices, and pre-processing operations such as denoising, filtering and distortion correction are performed, which significantly improves the integrity and accuracy of the data and lays a solid foundation for subsequent data fusion; multi-source data fusion and lightweight processing realize the efficient fusion of point cloud data, panoramic data and IoT perception data, and effectively reduces the cost of data storage and transmission; the variable node octree algorithm is introduced to construct hierarchical details and data slices for point cloud data, and the point cloud data, panoramic images and IoT perception data are combined for data fusion spatial database storage, which can integrate multiple data sources, provide comprehensive spatial perception information, and realize efficient spatial perception data storage; based on the multi-source data spatial query method and the real-time data interaction between the spatial database and the Web terminal, the real-time real-scene three-dimensional rendering of the spatial perception data model is realized, which improves the interactivity and real-time performance of data application. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0088] Figure 1 is a flow chart of the method of the present invention;

[0089] Figure 2 It is a system modular structure diagram of the present invention. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0091] like Figure 1 FIG. 1 is an embodiment of the present invention, which provides a spatially aware data storage method for multi-source data fusion, comprising the following steps:

[0092] S1: Collect point cloud data, panoramic data and IoT perception data through a variety of sensor devices;

[0093] Multi-source data acquisition is the foundation of the entire system. Acquiring comprehensive data through different types of sensors can provide a rich information basis for subsequent processing and analysis.

[0094] S2: Preprocessing the point cloud data, panoramic data and IoT sensing data respectively;

[0095] Different types of data have different characteristics and noise, and require targeted preprocessing, such as time synchronization, denoising, correction, etc., to ensure data consistency and accuracy, improve data quality, and lay the foundation for subsequent data fusion and analysis.

[0096] In one embodiment, step S2 specifically includes:

[0097] S21: Preprocessing the point cloud data, including:

[0098] Time synchronization: Use GPS timestamps or system timestamps to synchronize point cloud data from different devices, and use linear interpolation to estimate missing data points at the specified timestamp. The linear interpolation formula is:

[0099] ;

[0100] In the formula, and All timestamps are known , The point cloud coordinates under, t is the target timestamp, is the point cloud coordinate at the target timestamp;

[0101] Scale analysis:

[0102] For point cloud data from different devices or acquired under different conditions, select objects with known sizes or use the calibration information provided by the device to estimate the scale;

[0103] Determine the average scale or median scale of all point cloud data as the standard scale;

[0104] Calculate the scaling factor of each point cloud relative to the standard scale, and apply the scaling factor to adjust the scale of the point cloud;

[0105] Spatial alignment: Select a reference point cloud as the benchmark, use the iterative closest point algorithm to rotate and translate the other point cloud data outside the reference point cloud, and minimize the objective function ,in and are the matching point pairs, n is the matching point pair index; R is the rotation matrix, T is the translation vector; N is the number of matching point pairs;

[0106] Point cloud denoising: Apply statistical filters to remove isolated points in the point cloud data based on the average distance of each point's neighbors;

[0107] Point cloud filtering: Smoothing point cloud data by moving least squares method;

[0108] Distortion Correction:

[0109] The fast polynomial distortion model is used to remove the distortion in the point cloud data, specifically:

[0110] Before system deployment, use a calibration plate or known structure to obtain the distortion coefficient, integrate the correction formula into the data stream, and use multi-core processors for parallel calculation. The correction formula is:

[0111] ;

[0112] In the formula, x and y are the original coordinates, , is the coordinate after distortion correction; r is the distance from the point to the center of the image, k1, k2, k3 are all radial distortion coefficients;

[0113] For devices with known distortion patterns, a lookup table (LUT) is pre-calculated and fast correction is performed by looking up the table in real-time applications.

[0114] Point cloud data usually contains a lot of noise and distortion. These preprocessing steps can improve the accuracy and reliability of point cloud data and provide high-quality point cloud data for subsequent spatial alignment and data fusion.

[0115] S22: Preprocessing the panoramic data, including:

[0116] Image denoising: Apply bilateral filter or median filter to remove image noise;

[0117] Image correction: Use camera correction parameters to adjust the position of each pixel according to the radial and tangential distortion coefficients to eliminate optical distortion;

[0118] Resolution matching: Use bicubic interpolation for image scaling and adjust the image resolution to ensure that it matches the point cloud data. The formula is:

[0119] ;

[0120] In the formula, is the original image; , b is the original image The index in is used to traverse all pixels in the original image; Original image Index , b related pixel coordinates; is the resampled image, , is the new pixel coordinate in the resampled image; h is the interpolation kernel function;

[0121] Image stitching: Use the SIFT algorithm to detect feature points and the RANSAC algorithm to estimate the transformation and stitch the images into panoramic data;

[0122] Color consistency adjustment: Apply color balancing technology based on histogram matching to perform color balancing on the stitched panoramic data to eliminate image stitching gaps;

[0123] Image enhancement: Apply the limited contrast adaptive histogram equalization algorithm to perform local contrast enhancement.

[0124] Panoramic data requires high-quality image processing technology to eliminate noise and distortion, ensure image coherence and consistency, provide high-quality panoramic image data, and lay the foundation for subsequent data fusion and spatial rendering.

[0125] S23: Preprocessing the IoT sensing data, including:

[0126] Sensor data parsing: Parsing raw sensor data into a structured format;

[0127] Time synchronization: Timestamp each piece of IoT sensing data. If the data comes from different sensors and the timestamps are different, they are corrected according to the server time or GPS time, and synchronized by interpolation or alignment to the nearest integer second.

[0128] Normalization: Normalize the IoT sensing data and scale all data to the [0,1] range;

[0129] Data association: According to timestamp and location information, IoT sensing data is associated with objects in point cloud data and panoramic data to establish a mapping relationship between multi-source data.

[0130] IoT perception data usually comes from multiple sensors, and the time and space information may be inconsistent. It needs to be synchronized and normalized to ensure the consistency of IoT perception data and provide a basis for the fusion of multi-source data.

[0131] S3: Fuse the pre-processed point cloud data, panoramic data and IoT perception data into spatial perception data, and perform lightweight processing on the spatial perception data to generate a unified spatial perception data model;

[0132] Specifically, the implementation steps of the fusion processing and lightweight processing include:

[0133] S31: Use spatial reference coordinate conversion and time difference technology to perform time synchronization and spatial alignment on the pre-processed point cloud data, panoramic data and IoT perception data;

[0134] Spatial Reference is the skeleton framework of GIS data. It is a coordinate system used to store the coordinate attributes of each element class and raster dataset. It can locate our data to the corresponding location and provide accurate coordinates for each point in the map. Different types of data may be inconsistent in time and space. Time synchronization and spatial alignment can ensure data consistency, provide a unified time and space framework for the fusion of multi-source data, and ensure that subsequent data processing steps can be operated based on a consistent reference system.

[0135] S32: Use the curvature-improved farthest point sampling method to perform semantic segmentation on point cloud data, and use the improved Go-ICP algorithm to perform point cloud structure fitting in multiple time and space;

[0136] The curvature-improved farthest point sampling algorithm is an existing technology and will not be described in detail here. Meaningful features are extracted through semantic segmentation, the accuracy of point cloud data is improved through structural fitting, and high-quality point cloud data is extracted and fitted, providing a reliable data foundation for subsequent semantic label creation and data fusion;

[0137] Furthermore, the improved Go-ICP algorithm includes:

[0138] Nested BnB search: decompose the rotation and translation search into two independent levels. First, search for rotation parameters, and then search for translation parameters based on the determined rotation parameters. This improves the search efficiency, speeds up the search process, and promotes the overall point cloud registration process.

[0139] Registration error: The registration error is calculated by comparing the distances between the corresponding closest points in the source point cloud and the target point cloud, which serves as the basis for evaluating the registration accuracy;

[0140] Initial cube setting: Set the minimum search space cube, set the initial estimated pose as the side length, as the starting search range of the search algorithm;

[0141] Define uncertainty radius: Define an uncertainty radius during the search process to limit the search space, reduce unnecessary searches, improve algorithm efficiency, and ensure that the algorithm can adapt to point cloud data of different qualities;

[0142] Iterative optimization: Use iterative optimization pose to obtain the closest point pairs between the source point cloud and the target point cloud, and use these closest point pairs to update the pose of the source point cloud. The formula is:

[0143] ;

[0144] In the formula, It’s a new posture. It's the old position. is the transformation matrix calculated based on the nearest point pair information; K is the pose optimization coefficient, which is used to adjust the magnitude of the transformation;

[0145] Weight update: Update the point pair weights according to the distance and angle difference between the point pairs to enhance the robustness of the algorithm to point cloud data with different resolutions and distortions. The formula is:

[0146] cos( );

[0147] In the formula, is the weight of the point pair (i, j), d is the Euclidean distance between the point pairs (i, j), is the angle difference between the point pair (i, j), and It is an iterative optimization coefficient used to adjust the impact of distance and angle differences on point pair weights;

[0148] Convergence judgment: Set the registration error threshold and iteration limit. When the registration error during the iteration process is less than the preset registration error threshold or the number of iterations reaches the iteration limit, the iteration is stopped and the registration process is considered to have converged.

[0149] S33: Create a unified set of semantic labels across all data sources, covering all entities and attributes;

[0150] Provide consistent semantic labels for different data sources so that they can be accurately matched and parsed during fusion, and provide a consistent semantic framework to provide a basis for the fusion and analysis of multi-source data.

[0151] S34: Use NLP technology to identify entities in text, parse synonyms of entities, and establish a mapping table to map the identified and parsed entities into a unified semantic framework; Use natural language processing technology to identify and parse entities in text, improve the semantic consistency of data, ensure that entities in all data sources can be accurately identified and matched, and improve the accuracy of data fusion;

[0152] S35: Use the pre-trained BERT model to understand and map semantic tags and attributes; the BERT model has a strong semantic understanding ability, which can parse and map semantic tags and attributes more accurately, improve the accuracy of semantic tag and attribute mapping, and provide reliable semantic information for subsequent data integration and analysis;

[0153] S36: Input the spatiotemporally aligned and semantically matched point cloud data, panoramic data, and IoT sensing data into the multidimensional data integration model, and apply the Bayesian network to infer possible missing data or resolve data conflicts;

[0154] S37: Generative adversarial networks and variational autoencoders are used to fit point clouds and complete incomplete panoramic images at multiple spatiotemporal angles, and to perform probability distribution modeling on the missing parts. Generative adversarial networks and variational autoencoders can effectively complete missing data, improve data integrity and quality, provide complete and consistent spatial perception data, and provide a basis for subsequent standardization and optimized compression.

[0155] S38: Generate standardized spatial perception data with spatiotemporal fusion using the same semantic mapping method; standardized spatial perception data contributes to data consistency and availability, provides a standardized data model, and provides a consistent reference for subsequent data storage and query;

[0156] S39: Extract the features of spatial perception data through principal component analysis or autoencoder, optimize and compress the spatial perception data through differential pulse code modulation technology, encode it into a lightweight spatial perception data model, and import it into the spatial database; optimized compression and lightweight processing can improve the efficiency of data storage and transmission, reduce storage space and bandwidth requirements, generate efficient lightweight data models, and improve the storage and transmission efficiency of the system.

[0157] S4: Use variable node octree algorithm to construct hierarchical details and slice data of point cloud data;

[0158] Hierarchical detail construction and data slicing help improve the efficiency of data processing and rendering, adapt to the user's perspective and operation requirements, provide efficient data processing and rendering mechanisms, and enhance the real-time response capabilities of the system;

[0159] In one embodiment, step S4 comprises:

[0160] S41: Determine the root node boundary of the octree according to the minimum and maximum coordinate values ​​of the point cloud data;

[0161] S42: recursively assigning each point in the point cloud data to an octree node until a preset tree depth is reached or the number of points in the node reaches an upper limit;

[0162] S43: Each octree node represents a data slice, and the detail level of the data slice is automatically adjusted according to the user's perspective, specifically: capturing the user's zoom operation or view change event, and selecting the octree depth according to the user's zoom level or view distance; analyzing the user's current viewpoint position and line of sight direction in real time, selecting the octree node aligned with the user's line of sight direction, and preferentially loading the point cloud data in the octree node.

[0163] S5: Encode the spatial identity of the point cloud data and panoramic data, establish a spatiotemporal index, and store the spatial perception data model in the spatial database in the same geographic coordinate system;

[0164] Spatial identity encoding:

[0165] Spatial identity encoding is a technique used to assign a unique identifier to an object or location in space, which facilitates fast indexing and retrieval of spatial objects in a database or virtual environment. This encoding technique is particularly important in 3D modeling and virtual reality because it allows systems to efficiently process and interact with large amounts of spatial data.

[0166] Spatiotemporal index:

[0167] A spatiotemporal index is a data structure that not only considers spatial location but also combines time information to optimize the query and storage of data with both temporal and spatial dimensions. In 3D modeling and virtual reality technology, spatiotemporal indexes can be used to manage and query spatial data that changes over time, such as tracking the path of a moving object or simulating historical scene changes.

[0168] Spatial identity coding and spatiotemporal indexing facilitate rapid retrieval and access of data, improve the storage and processing efficiency of the system, provide an efficient data storage and retrieval mechanism, and support the system's real-time data interaction and analysis needs.

[0169] Specifically, step S5 includes:

[0170] S51: Use Geohash or S2 geometry libraries to assign unique spatial identity codes to point cloud and panoramic data;

[0171] S52: Combine timestamps and spatial encodings to build a composite spatiotemporal index, store the spatially aware data model in a spatial database, and optimize data storage and access through the spatiotemporal index.

[0172] S6: Based on the multi-source data spatial query method and the real-time data interaction between the spatial database and the Web end, the spatial perception data model is rendered in real time in real-time 3D.

[0173] Furthermore, the spatial perception data model is subjected to real-time three-dimensional rendering, specifically including:

[0174] S61: Acquire the target to be rendered according to the current view cone coordinates, and simultaneously acquire the view cone space geometry information of the last rendered scene;

[0175] S62: Calculate the newly added field of view range information according to the current view cone coordinates and the view cone space geometry information of the previous scene;

[0176] S63: Sending a data call request including the newly added field of view information to the server;

[0177] S64: After receiving the data call request, the server performs a geometric interleaving query on the point cloud data in the spatial database according to the newly added field of view information;

[0178] S65: After the query is completed, the server sends the newly added spatiotemporal index to the client;

[0179] S66: The client queries the corresponding point cloud data according to the newly added spatiotemporal index, renders the target, and maps the corresponding panoramic data according to the spatiotemporal index of the queried point cloud data, and preloads the panoramic data.

[0180] like Figure 2 FIG. 1 is another embodiment of the present invention, which provides a spatial perception data storage system for multi-source data fusion, based on the spatial perception data storage method for multi-source data fusion as described above, including:

[0181] The data acquisition module is used to collect point cloud data, panoramic data and IoT perception data through a variety of devices or sensors;

[0182] The data preprocessing module is used to preprocess the point cloud data, panoramic data and IoT sensing data respectively;

[0183] The data fusion module is used to fuse the pre-processed point cloud data, panoramic data and IoT perception data into spatial perception data, and to perform lightweight processing on the spatial perception data to generate a unified spatial perception data model;

[0184] Octree module, used to construct hierarchical details and slice data of point cloud data using variable node octree algorithm;

[0185] An index encoding module, used to perform spatial identity encoding on point cloud data and panoramic data, and to establish a spatiotemporal index, and to store the spatial perception data model in a spatial database in the same geographic coordinate system;

[0186] The real-scene rendering module is used to perform real-scene 3D rendering of the spatial perception data model in real time based on the multi-source data spatial query method and the real-time data interaction between the spatial database and the Web terminal.

[0187] In summary, the present invention collects point cloud data, panoramic data and IoT perception data respectively through a variety of sensor devices, and performs preprocessing operations such as denoising, filtering and distortion correction, which significantly improves the integrity and accuracy of the data and lays a solid foundation for subsequent data fusion; multi-source data fusion and lightweight processing realize the efficient fusion of point cloud data, panoramic data and IoT perception data, and effectively reduces the cost of data storage and transmission; introduces a variable node octree algorithm to construct hierarchical details and data slicing of point cloud data, and combines point cloud data, panoramic images and IoT perception data for data fusion spatial database storage, which can integrate multiple data sources, provide comprehensive spatial perception information, and realize efficient spatial perception data storage; based on the multi-source data spatial query method and the real-time data interaction between the spatial database and the Web terminal, the real-time real-scene three-dimensional rendering of the spatial perception data model is realized, which improves the interactivity and real-time performance of data application.

[0188] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A spatial perception data storage method for multi-source data fusion, characterized in that: The method comprises: Collect point cloud data, panoramic data and IoT perception data through multiple devices or sensors; Preprocessing the point cloud data, panoramic data and IoT sensing data respectively; The IoT sensing data is preprocessed, including: Sensor data parsing: Parsing raw sensor data into a structured format; Time synchronization: Timestamp each piece of IoT sensing data. If the data comes from different sensors and the timestamps are different, they are corrected according to the server time or GPS time, and synchronized by interpolation or alignment to the nearest integer second. Normalization: Normalize the IoT sensing data and scale all data to the [0,1] range; Data association: According to timestamp and location information, IoT sensing data is associated with objects in point cloud data and panoramic data to establish a mapping relationship between multi-source data; The pre-processed point cloud data, panoramic data and IoT perception data are fused into spatial perception data, and the spatial perception data is lightweight processed to generate a unified spatial perception data model; The steps for implementing the fusion processing and lightweight processing include: Use spatial reference coordinate conversion and time difference technology to perform time synchronization and spatial alignment on the pre-processed point cloud data, panoramic data and IoT perception data; The farthest point sampling method with improved curvature is used to perform semantic segmentation on point cloud data, and the improved Go-ICP algorithm is used to fit the point cloud structure in multiple time and space. The improved Go-ICP algorithm comprises: Nested BnB search: decomposes the search for rotation and translation into two independent levels, first searching for rotation parameters, and then searching for translation parameters based on the determined rotation parameters; Registration error: The registration error is calculated by comparing the distances between the corresponding closest points in the source point cloud and the target point cloud; Initial cube setting: Set the minimum search space cube and set the initial estimated pose as the side length; Uncertainty radius definition: Define an uncertainty radius during the search process to limit the search space; Iterative optimization: Use iterative optimization pose to obtain the closest point pair between the source point cloud and the target point cloud, and update the pose of the source point cloud. The formula is: ； In the formula, It’s a new posture. It's the old position. is the transformation matrix calculated based on the nearest point pair information; K is the pose optimization coefficient; Weight update: Update the point pair weights according to the distance and angle difference between the point pairs. The formula is: cos( ); In the formula, is the weight of the point pair (i, j), d is the Euclidean distance between the point pairs (i, j), is the angle difference between the point pair (i, j), and is the iterative optimization coefficient; Convergence judgment: Set the registration error threshold and iteration limit. When the registration error during the iteration process is less than the preset registration error threshold or the number of iterations reaches the iteration limit, the iteration is stopped and the registration process is considered to have converged. Create a unified set of semantic labels across all data sources, covering all entities and attributes; Use NLP technology to identify entities in text, parse synonyms of entities, and build a mapping table to map the identified and parsed entities into a unified semantic framework; Leverage the pre-trained BERT model to understand and map semantic labels and attributes; Input the spatiotemporally aligned and semantically matched point cloud data, panoramic data, and IoT sensing data into the multidimensional data integration model, and apply the Bayesian network to infer possible missing data or resolve data conflicts; Generative adversarial networks and variational autoencoder technologies are used to fit point clouds and complete incomplete panoramic images at multiple time and space angles, and to perform probability distribution modeling for the missing parts; Generate standardized spatial perception data with spatiotemporal fusion by using the same semantic mapping method; The features of spatial perception data are extracted through principal component analysis or autoencoder, and the spatial perception data is optimized and compressed through differential pulse code modulation technology, encoded into a lightweight spatial perception data model, and imported into the spatial database; Use variable node octree algorithm to construct hierarchical details and slice data of point cloud data; Encoding the spatial identity of the point cloud data and the panoramic data, establishing a spatiotemporal index, and storing the spatial perception data model in a spatial database in the same geographic coordinate system; Based on a multi-source data spatial query method and real-time data interaction between a spatial database and a Web terminal, the spatial perception data model is subjected to real-time three-dimensional rendering.

2. The spatial perception data storage method of multi-source data fusion according to claim 1 is characterized in that: The point cloud data is preprocessed, including: Time synchronization: Use GPS timestamps or system timestamps to synchronize point cloud data from different devices, and use linear interpolation to estimate missing data points at the specified timestamp. The linear interpolation formula is: ； In the formula, and All timestamps are known , The point cloud coordinates under, t is the target timestamp, is the point cloud coordinate at the target timestamp; Scale analysis: For point cloud data from different devices or acquired under different conditions, select objects of known size or use the calibration information provided by the device to estimate the scale; Determine the average scale or median scale of all point cloud data as the standard scale; Calculating a scaling factor of each point cloud relative to a standard scale, and applying the scaling factor to adjust the scale of the point cloud; Spatial alignment: Select a reference point cloud as the benchmark, use the iterative closest point algorithm to rotate and translate the other point cloud data outside the reference point cloud, and minimize the objective function ,in and are the matching point pairs, n is the matching point pair index; R is the rotation matrix, T is the translation vector; N is the number of matching point pairs; Point cloud denoising: Apply statistical filters to remove isolated points in the point cloud data based on the average distance of each point's neighbors; Point cloud filtering: Smoothing point cloud data by moving least squares method; Distortion correction: A fast polynomial distortion model is used to remove distortion in point cloud data. Specifically: Before system deployment, use a calibration plate or known structure to obtain the distortion coefficient, integrate the correction formula into the data stream, and use multi-core processors for parallel calculation. The correction formula is: ； In the formula, x and y are the original coordinates, , is the coordinate after distortion correction; r is the distance from the point to the center of the image, k1, k2, k3 are all radial distortion coefficients; For devices with known distortion patterns, a lookup table (LUT) is pre-calculated and fast correction is performed by looking up the table in real-time applications.

3. The spatial perception data storage method of multi-source data fusion according to claim 1 is characterized in that: The panoramic data is preprocessed, including: Image denoising: Apply bilateral filter or median filter to remove image noise; Image correction: Use camera correction parameters to adjust the position of each pixel according to the radial and tangential distortion coefficients to eliminate optical distortion; Resolution matching: Use bicubic interpolation for image scaling and adjust the image resolution to ensure that it matches the point cloud data. The formula is: ； In the formula, is the original image; , b is the original image The index in is used to traverse all pixels in the original image; Original image Index , b related pixel coordinates; is the resampled image, , is the new pixel coordinate in the resampled image; h is the interpolation kernel function; Image stitching: Use the SIFT algorithm to detect feature points and the RANSAC algorithm to estimate the transformation and stitch the images into panoramic data; Color consistency adjustment: Apply color balancing technology based on histogram matching to perform color balancing on the stitched panoramic data to eliminate image stitching gaps; Image enhancement: Apply the limited contrast adaptive histogram equalization algorithm to perform local contrast enhancement.

4. The spatial perception data storage method of multi-source data fusion according to claim 1 is characterized in that: The method of using the variable node octree algorithm to construct hierarchical details and slice data on point cloud data includes: Determine the root node boundary of the octree according to the minimum and maximum coordinate values ​​of the point cloud data; Recursively assign each point in the point cloud data to an octree node until the preset tree depth is reached or the number of points in the node reaches an upper limit; Each octree node represents a data slice, and the detail level of the data slice is automatically adjusted according to the user's perspective, specifically: capturing the user's zoom operation or view change event, selecting the octree depth according to the user's zoom level or view distance; analyzing the user's current viewpoint position and line of sight direction in real time, selecting the octree node aligned with the user's line of sight direction, and preferentially loading the point cloud data in the octree node.

5. The spatial perception data storage method of multi-source data fusion according to claim 1 is characterized in that: The step of encoding the spatial identity of the point cloud data and the panoramic data, establishing a spatiotemporal index, and storing the spatial perception data model in the spatial database in the same geographic coordinate system specifically includes: Use Geohash or S2 geometry libraries to assign unique spatial identity codes to point cloud data and panoramic data; A composite spatiotemporal index is constructed by combining the timestamp and the spatial identity code, the spatially aware data model is stored in a spatial database, and the storage and access of data are optimized through the spatiotemporal index.

6. The spatial perception data storage method of multi-source data fusion according to claim 1 is characterized in that: The real-time three-dimensional rendering of the spatial perception data model specifically includes: Get the target to be rendered according to the current view frustum coordinates, and at the same time get the view frustum space geometry information of the previous rendering scene; Calculate the newly added field of view range information based on the current view frustum coordinates and the view frustum space geometry information of the previous scene; Send a data call request containing the newly added field of view information to the server; After receiving the data call request, the server performs geometric interleaving query on the point cloud data in the spatial database based on the newly added field of view information; After the query is completed, the server sends the newly added spatiotemporal index to the client; The client queries the corresponding point cloud data according to the newly added spatiotemporal index, renders the target, and maps the corresponding panoramic data according to the spatiotemporal index of the queried point cloud data, and preloads the panoramic data.

7. A spatial perception data storage system for multi-source data fusion, based on a spatial perception data storage method for multi-source data fusion as claimed in any one of claims 1 to 6, characterized in that: The system comprises: The data acquisition module is used to collect point cloud data, panoramic data and IoT perception data through a variety of devices or sensors; A data preprocessing module, used to perform preprocessing operations on the point cloud data, panoramic data and IoT sensing data respectively; A data fusion module is used to fuse the pre-processed point cloud data, panoramic data and IoT perception data into spatial perception data, and to perform lightweight processing on the spatial perception data to generate a unified spatial perception data model; Octree module, used to construct hierarchical details and slice data of point cloud data using variable node octree algorithm; An index encoding module, used to perform spatial identity encoding on point cloud data and panoramic data, and to establish a spatiotemporal index, and to store the spatial perception data model in a spatial database in the same geographic coordinate system; The real scene rendering module is used to perform real scene three-dimensional rendering of the spatial perception data model in real time based on a multi-source data spatial query method and real-time data interaction between a spatial database and a Web terminal.

Images