A method and system for multimodal data unification processing and visualization

Abstract

This invention relates to the fields of computer 3D visualization and virtual reality technology, and particularly to a method and system for unified processing and visualization of multimodal data. The method includes the following steps: S1, establishing a global spatial coordinate system and a unified temporal reference system, mapping multi-source heterogeneous data such as satellite imagery, vector maps, and 3D models to a 3D geospatial rendering environment to generate standardized spatiotemporal data resources; S2, constructing a spatial index model for the standardized spatiotemporal data resources based on an adaptive bounding box tree and scene graph data structure. In this invention, by constructing an adaptive spatial index model and performing view frustum culling to quickly filter invalid rendering objects, and in conjunction with a memory scheduling mechanism based on viewpoint motion prediction and LRU-K rules to achieve intelligent pre-fetching and timely release of data nodes, the high frame rate smoothness and resource utilization efficiency of large-scale scene roaming are ensured while significantly reducing the graphics pipeline load.

Description

Technical Field

[0001] This invention relates to the field of computer 3D visualization and virtual reality technology, and in particular to a method and system for unified processing and visualization of multimodal data. Background Technology

[0002] With the rapid development of technologies related to digital twin cities, smart earth, and metaverse, the means of collecting geospatial information data are becoming increasingly diverse and accurate. Spatiotemporal big data, encompassing high-resolution satellite remote sensing imagery, complex urban vector road networks, and refined building BIM models, is experiencing explosive growth. This necessitates that modern 3D visualization systems achieve seamless transitions and full-element displays from macroscopic global perspectives to microscopic indoor scenes within extremely large virtual geographic environments. To meet users' high-frequency interactive needs for real-time roaming, free scaling, and instant querying in massive, multi-source, heterogeneous data scenarios, rendering engines must possess extremely high data throughput and graphics processing efficiency. Under the constraints of limited computer memory resources and computing bandwidth, they must efficiently organize, real-time crop, and dynamically schedule hundreds of millions of geometric primitives and massive texture data, thereby constructing a large-scale immersive visualization platform that possesses both highly realistic visual effects and a smooth user experience.

[0003] However, existing 3D scene rendering technologies often rely on static conventional spatial partitioning structures and passive on-demand loading mechanisms when processing such large-scale data. They lack accurate prediction of the user's intention to move the viewpoint at high speed, which often leads to blank screens or obvious visual stuttering caused by data reading delays during fast roaming. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides a method and system for unified processing and visualization of multimodal data, aiming to improve the problem that existing 3D scene rendering technologies often rely on static conventional spatial partitioning structures and passive on-demand loading mechanisms when processing such large-scale data.

[0005] In a first aspect, the present invention provides the following technical solution: a method for unified processing and visualization of multimodal data, comprising the following steps:

[0006] S1. Establish a global spatial coordinate system and a unified time reference system, map multi-source heterogeneous data such as satellite imagery, vector maps and 3D models to the 3D geospatial rendering environment, and generate standardized spatiotemporal data resources.

[0007] S2. Based on the adaptive bounding box tree and scene graph data structure, construct a spatial index model for the standardized spatiotemporal data resources, perform collision detection and view frustum culling, and extract the set of scene data in the visible field of view from the current perspective.

[0008] S3. Calculate the viewpoint motion vector based on the user interaction trajectory, and use the LRU-K algorithm to pre-read and schedule the spatially adjacent data nodes of the visible scene data set to generate a pre-buffered rendering data queue.

[0009] S4. Using a depth buffer and a hybrid rendering pipeline, perform layered fusion rendering on the pre-buffered rendering data queue to generate an initial hybrid scene view containing two-dimensional layers and three-dimensional models.

[0010] S5. In response to the interaction command for the initial mixed scene view, call the preset parameter mapping rules to convert the two-dimensional vector map elements into three-dimensional rendering data objects and inject them into the rendering pipeline to obtain the updated multi-source fusion visualization result.

[0011] Preferably, in step S1, generating standardized spatiotemporal data resources specifically includes the following steps:

[0012] The CGCS2000 coordinate system is established as the global spatial coordinate system, and the unified time reference system is established.

[0013] The raster data of satellite imagery, the two-dimensional vector map elements of vector maps, and the grid data of three-dimensional models are acquired respectively.

[0014] Perform coordinate projection transformation and timestamp alignment on the raster data, the two-dimensional vector map elements, and the grid data to uniformly map them to the global spatial coordinate system and the unified time reference system;

[0015] Using a matching algorithm based on timestamps and spatial indexes, the aligned data is mounted to the corresponding level node of the three-dimensional geospatial rendering environment to obtain the standardized spatiotemporal data resources.

[0016] Preferably, in step S2, constructing a spatial index model for the standardized spatiotemporal data resources based on the adaptive bounding box tree and scene graph data structure specifically includes the following steps:

[0017] The various entity objects in the standardized spatiotemporal data resources are analyzed, and a scene graph data structure containing root nodes, transformation matrix nodes and leaf geometry nodes is constructed based on their entity attributes and logical relationships.

[0018] Calculate the bounding box of each leaf geometric node in the scene graph data structure, and generate the corresponding adaptive bounding box tree using a bottom-up aggregation strategy;

[0019] Establish bidirectional mapping pointers between the spatial nodes of the adaptive bounding box tree and the logical nodes of the scene graph data structure to generate a spatial index model that associates spatial locations with logical attributes.

[0020] Preferably, in step S2, performing collision detection and frustum culling specifically includes the following steps:

[0021] Obtain the camera parameters and projection matrix of the current viewpoint, and construct the view frustum space containing six clipping planes;

[0022] Traverse the spatial index model from the root node and perform an intersection test between the node bounding boxes of the adaptive bounding box tree and the spatial extent of the view frustum;

[0023] Node branches that are disjoint from the spatial range of the view frustum are excluded, nodes that intersect or contain each other are retained, and geometric information is extracted from the corresponding scene graph data structure to generate a set of viewable scene data.

[0024] Preferably, in step S3, the pre-fetching and memory scheduling of spatially adjacent data nodes in the visible scene data set using the LRU-K algorithm specifically includes the following steps:

[0025] Record the historical access timestamps of each data node in the visible field scene data set, and maintain an access history list containing K historical access records;

[0026] The predicted access probability of spatially adjacent data nodes within a preset range outside the visible field boundary is calculated based on the viewpoint motion vector of the current viewpoint, and the cache priority weight of each data node is calculated in combination with the access history list.

[0027] The adjacent data nodes with cache priority weights higher than the preset loading threshold are asynchronously loaded into video memory and added to the pre-buffered rendering data queue.

[0028] Monitor the video memory usage status. When the usage rate exceeds the preset usage threshold, retrieve and release the video memory space occupied by data nodes that meet the eviction conditions according to the LRU-K rule.

[0029] Preferably, in step S4, the layered fusion rendering of the pre-buffered rendering data queue using a depth buffer and a hybrid rendering pipeline specifically includes the following steps:

[0030] Initialize the blending pipeline, configure the depth test function and color blending mode, and reset the color buffer and depth buffer;

[0031] Based on the rendering layer priority, the pre-buffered rendering data queue is divided into a background terrain layer, a vector feature layer, and a 3D model layer;

[0032] The background terrain layer is drawn, and the depth values ​​of the corresponding fragments are written into the depth buffer to construct the depth reference of the basic scene.

[0033] The vector feature layer and the 3D model layer are drawn sequentially. The depth value of the fragment to be drawn is compared with the reference value in the depth buffer. Color mixing calculation is performed only on the fragments that pass the test.

[0034] After completing the fragment shading and blending of all levels of data, output the initial blended scene view after compositing.

[0035] Preferably, in step S5, the process of converting two-dimensional vector map features into three-dimensional rendering data objects and injecting them into the rendering pipeline by invoking preset parameter mapping rules specifically includes the following steps:

[0036] Listen for user operation events in response to the initial mixed scene view, and parse the interaction instructions to identify the selected two-dimensional vector map features and their attribute data;

[0037] Based on the geometric type and attribute characteristics of the two-dimensional vector map elements, the corresponding parameter mapping rules are retrieved from the preset style library. The rules define the conversion logic from two-dimensional planar coordinates to three-dimensional spatial coordinates and the appearance rendering parameters.

[0038] By executing the parameter mapping rules, the planar geometric data of the two-dimensional vector map features is reconstructed into a three-dimensional geometric mesh, and the attribute data is mapped into three-dimensional rendering attributes to generate a three-dimensional rendering data object.

[0039] The 3D rendering data object is dynamically attached to the list of nodes to be rendered in the hybrid rendering pipeline, triggering a scene redraw and refresh, and outputting a multi-source fusion visualization result.

[0040] Secondly, the present invention provides the following technical solution: a multimodal data unified processing and visualization system, the system comprising:

[0041] The data standardization mapping module is used to establish a global spatial coordinate system and a unified time reference system, and to map multi-source heterogeneous data such as satellite imagery, vector maps and 3D models to the 3D geospatial rendering environment to generate standardized spatiotemporal data resources.

[0042] The visible field scene extraction module is used to construct a spatial index model for the standardized spatiotemporal data resources based on the adaptive bounding box tree and scene graph data structure, perform collision detection and view frustum culling, and extract the visible field scene data set of the current viewpoint;

[0043] The pre-scheduled cache module is used to calculate the viewpoint motion vector based on the user interaction trajectory, and to use the LRU-K algorithm to pre-read and schedule the spatially adjacent data nodes of the visible scene data set to generate a pre-buffered rendering data queue.

[0044] The layered blending rendering module is used to perform layered blending rendering on the pre-buffered rendering data queue using a depth buffer and a blending rendering pipeline, generating an initial blended scene view containing two-dimensional layers and three-dimensional models.

[0045] The dynamic interaction synchronization module is used to respond to the interaction commands for the initial mixed scene view, call the preset parameter mapping rules to convert the two-dimensional vector map elements into three-dimensional rendering data objects and inject them into the rendering pipeline to obtain the updated multi-source fusion visualization results.

[0046] The present invention has the following beneficial effects:

[0047] 1. In this invention, an adaptive spatial indexing model is constructed and view frustum culling is performed to quickly filter invalid rendering objects. Combined with a memory scheduling mechanism based on viewpoint motion prediction and LRU-K rules, intelligent pre-fetching and timely release of data nodes are achieved. This significantly reduces the load on the graphics pipeline while ensuring high frame rate smoothness and resource utilization efficiency for large-scale scene roaming.

[0048] 2. In this invention, a unified spatiotemporal reference system is used to eliminate the benchmark differences of multi-source heterogeneous data. A layered hybrid rendering strategy based on depth buffer is adopted to strictly control the rendering priority and depth testing logic, which effectively solves the problems of depth conflict and occlusion disorder when two-dimensional vectors and three-dimensional models are mixed, and ensures the visual correctness of multimodal data fusion display in the same scene.

[0049] 3. In this invention, the selected two-dimensional vector elements are reconstructed into three-dimensional rendering objects in real time according to the parameter mapping rules invoked by the user interaction instructions. This realizes the on-demand generation and dynamic mounting of abstract planar data into concrete spatial entities, which not only enriches the information interaction dimension of the visualization scene, but also avoids the memory redundancy and performance loss caused by fully loading high-precision three-dimensional models. Attached Figure Description

[0050] Figure 1 This is a flowchart of a method for unified processing and visualization of multimodal data proposed in this invention;

[0051] Figure 2 This is a diagram of the overall logical architecture proposed in this invention;

[0052] Figure 3 This is an architecture diagram of a multimodal data unified processing and visualization system proposed in this invention. Detailed Implementation

[0053] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Example 1:

[0055] In the first embodiment of the present invention, the present invention provides a method for unified processing and visualization of multimodal data, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0056] S1. Establish a global spatial coordinate system and a unified time reference system, map multi-source heterogeneous data such as satellite imagery, vector maps and 3D models to the 3D geospatial rendering environment, and generate standardized spatiotemporal data resources.

[0057] Furthermore, in S1, generating standardized spatiotemporal data resources specifically includes the following steps:

[0058] The CGCS2000 coordinate system was established as the global spatial coordinate system, and a unified time reference system was also established.

[0059] The raster data of satellite imagery, the two-dimensional vector map elements of vector maps, and the grid data of three-dimensional models are acquired respectively.

[0060] Perform coordinate projection transformation and timestamp alignment on raster data, 2D vector map features and grid data to uniformly map them to the global spatial coordinate system and a unified time reference system;

[0061] By using a matching algorithm based on timestamps and spatial indexes, the aligned data is mounted to the corresponding hierarchical nodes in the 3D geospatial rendering environment to obtain standardized spatiotemporal data resources.

[0062] Specifically, the rendering engine first establishes the China Geodetic Coordinate System 2000 (CGCS2000) as the globally unique spatial reference datum in its global configuration, and sets Coordinated Universal Time (UTC) as the unified time reference system. The system utilizes the GDAL geospatial data abstraction library and the Assimp model import library to read multi-source heterogeneous data stored in the disk array. This includes GeoTIFF format satellite image raster data with geolocation information and Shapefile format vector data containing point, line, and polygon geometric information. Figure 2 3D vector map elements, and OSGB format oblique photogrammetric 3D model mesh data containing vertex and texture information.

[0063] For the various heterogeneous data types read, the system parses their metadata header files one by one to identify the original coordinate system. For data whose original coordinate system is not CGCS2000, the system executes a coordinate transformation algorithm based on a seven-parameter model to uniformly map their geometric positions to the global spatial coordinate system. Taking the conversion of geographic coordinates in the geodetic coordinate system to geocentric rectangular coordinates used by the rendering engine as an example, the input parameter is defined as any geometric location point in the original data. This point contains geodetic longitude. Earth's latitude and Earth High Define the semi-major axis of the CGCS2000 reference ellipsoid. It is 6,378,137.0 meters, with a flattening ratio of The value is 1 / 298.257222101. First, calculate the square of the first eccentricity. The calculation formula is: Next, calculate the radius of curvature of the zonal circle at that latitude. The calculation formula is: Finally, the output result is the mapped three-dimensional Cartesian space coordinates. The calculation formula is as follows:

[0064] ;

[0065] ;

[0066] ;

[0067] The system iterates through the center point and vector map of every pixel in the satellite image raster data. Figure 2 Each node of the 3D vector map element and each vertex of the 3D model grid data are substituted into the above formula for calculation, thereby unifying the spatial position of all data into a Cartesian coordinate system with the Earth's center as the origin.

[0068] While completing spatial mapping, the system performs unified processing on the temporal attributes of the data. The system parses the acquisition time field from various data metadata, converts it into a long integer value in UTC timestamp format conforming to the ISO 8601 standard using a time parsing function, and then executes a matching algorithm based on timestamps and spatial indices to mount the aligned data to scene graph nodes in the 3D geospatial rendering environment. This algorithm defines the input as the data resource object to be mounted. and target hierarchy nodes in the rendering environment Define data resource objects Minimum enclosing box in space The minimum value in the three axes is The maximum value is Define target level nodes The minimum value of the spatial range in the three axes is The maximum value is Define the timestamp of the data as The time span of the node is the start time. Until the end time The system determines a successful match based on the following conditions: the spatial ranges intersect and the time ranges are within a certain range, meaning the following logical relationships must be satisfied simultaneously:

[0069] ;

[0070] ;

[0071] ;

[0072] ;

[0073] When all the above conditions are met, the system will store the data resource object. Mark as node The sub-associated objects are identified, and pointer indexes are created in memory to complete the data mounting. If the conditions are not met after traversing all nodes at the current level, a new node is created according to the quadtree space partitioning rules, and the above checks are repeated.

[0074] By employing rigorous mathematical coordinate transformation and logical matching based on axis-aligned bounding boxes (AABB), spatial misalignment caused by different coordinate system definitions in multi-source data and temporal disorder caused by differences in time formats are eliminated. This provides a standardized data foundation with accurate geometric position and consistent temporal logic for subsequent frustum culling, collision detection, and hybrid rendering.

[0075] S2. Based on the adaptive bounding box tree and scene graph data structure, a spatial index model is constructed for standardized spatiotemporal data resources, collision detection and view frustum culling are performed, and the set of scene data in the visible field of view of the current view is extracted.

[0076] Furthermore, in S2, the construction of a spatial index model for standardized spatiotemporal data resources based on adaptive bounding box trees and scene graph data structures specifically includes the following steps:

[0077] Analyze various entity objects in standardized spatiotemporal data resources, and construct a scene graph data structure containing root nodes, transformation matrix nodes, and leaf geometry nodes based on their entity attributes and logical relationships;

[0078] Calculate the bounding box of each leaf geometric node in the scene graph data structure, and generate the corresponding adaptive bounding box tree using a bottom-up aggregation strategy;

[0079] Establish bidirectional mapping pointers between spatial nodes of the adaptive bounding box tree and logical nodes of the scene graph data structure to generate a spatial index model that associates spatial locations with logical attributes.

[0080] Furthermore, in S2, performing collision detection and frustum culling specifically includes the following steps:

[0081] Obtain the camera parameters and projection matrix of the current viewpoint, and construct the view frustum space containing six clipping planes;

[0082] Traverse the spatial index model from the root node and perform an intersection test between the node bounding boxes of the adaptive bounding box tree and the view frustum spatial extent;

[0083] Remove nodes that are disjoint from the view frustum space, retain nodes that intersect or contain each other, and extract the geometric information from the corresponding scene graph data structure to generate a viewable scene data set.

[0084] Specifically, after constructing the standardized spatiotemporal data resources, the system first parses the standardized data list in memory, initializes the scene graph management module, and instantiates logical node objects according to the type attributes of entity objects. It then constructs a hierarchical scene graph data structure starting from the root node and including transformation matrix nodes and leaf geometric nodes. For each leaf geometric node, the system traverses all its contained geometric vertex data, calculates and generates the axis-aligned bounding box (AABB) of that node, and defines the set of vertices contained in the leaf node as follows: For the set of the first vertices Its coordinates are Then the coordinates of the minimum point of the bounding box of the leaf node. and the coordinates of the maximum point The calculation formula is as follows:

[0085] ;

[0086] ;

[0087] After obtaining the bounding boxes of all leaf nodes, the system uses a bottom-up aggregation strategy to construct an adaptive bounding box tree. For any parent node, the system traverses the bounding box set of all its child nodes and calculates the bounding box range of the parent node. The calculation logic is to take the minimum value among the minimum values ​​of the bounding boxes of all child nodes in each axis as the minimum point of the parent node, and the maximum value among the maximum values ​​as the maximum point of the parent node. After construction, the system stores memory address pointers pointing to the corresponding logical nodes of the scene graph in the spatial node objects of the adaptive bounding box tree, and records the references of the corresponding spatial index nodes in the attribute fields of the scene graph nodes, thereby establishing bidirectional mapping pointers and generating a spatial index model that associates spatial location with logical attributes.

[0088] Subsequently, the system enters the preprocessing stage of the rendering loop, performing collision detection and frustum culling. The system first reads the view matrix of the current rendering camera. and projection matrix Multiplying the two matrices yields the view projection matrix. The calculation formula is: Let matrix It is a 4x4 matrix, the first... Line 1 The elements of a column are denoted as The system is based on the matrix Extract the six clipping planes of the view frustum; the general equation of the planes is: Taking the left cutting plane as an example, the formula for calculating its coefficient is:

[0089] ;

[0090] ;

[0091] ;

[0092] ;

[0093] Similarly, the other five planes—right, bottom, top, near, and far—can be extracted. After extraction, the system obtains the normal vector for each plane. Perform normalization to ensure that the normal vector length is 1.

[0094] The system traverses the spatial index model starting from the root node, performing an intersection test between the bounding box and the view frustum for each node. For any plane of the given node's bounding box and view frustum, the system finds the positive vertex on the bounding box that is furthest from the direction of the plane's normal vector. Let the minimum point of the bounding box be... The maximum point is The plane normal vector is Then the positive vertex The rules for selecting coordinate components are as follows: If but Pick otherwise Pick ,like but Pick otherwise Pick ,like but Pick otherwise Pick The system calculates the directed distance from the positive vertex to the plane. :

[0095] ;

[0096] If for any plane of the view frustum, the calculated distance is... If the value is less than 0, the bounding box of the node is determined to be completely outside the view frustum. The system marks the node and all its child nodes as disjoint and removes them directly. If the node passes the test of all planes, it is determined to be intersecting or contained, and the system retains it. For the retained leaf nodes, the system uses bidirectional mapping pointers to index the scene graph data structure, extracts the corresponding geometric model data and material properties, and finally generates the set of visible scene data under the current view.

[0097] By constructing an efficient spatial index structure and combining it with a view frustum culling algorithm, a large amount of invalid data outside the viewport can be quickly filtered out, significantly reducing the geometric processing load of the rendering pipeline and ensuring real-time rendering frame rate in large-scale multimodal data scenarios.

[0098] S3. Calculate the viewpoint motion vector based on the user interaction trajectory, and use the LRU-K algorithm to pre-read and schedule the spatially adjacent data nodes of the visible scene data set to generate a pre-buffered rendering data queue.

[0099] Furthermore, in S3, the LRU-K algorithm is used to pre-fetch and schedule memory for spatially adjacent data nodes in the visible scene dataset, specifically including the following steps:

[0100] Record the historical access timestamps of each data node in the visible scene data set, and maintain an access history list containing K historical access records;

[0101] The predicted access probability of spatially adjacent data nodes within a preset range outside the visible field of view is calculated based on the viewpoint motion vector of the current viewpoint, and the cache priority weight of each data node is calculated in combination with the access history list.

[0102] Asynchronously load adjacent data nodes with cache priority weights higher than the preset loading threshold into video memory and add them to the pre-buffered rendering data queue.

[0103] Monitor the video memory usage status. When the usage rate exceeds the preset usage threshold, retrieve and release the video memory space occupied by data nodes that meet the eviction conditions according to the LRU-K rule.

[0104] Specifically, after generating the visual field scene data set, the system tracks user interactions in real time, obtains the camera spatial coordinates of the current frame and the previous frame by listening to the rendering engine's frame callback events, and defines the current moment as... The current viewpoint coordinates are The viewpoint coordinates at the previous frame were Both are included The system calculates the viewpoint motion vector from the three-dimensional vector of the components. The calculation formula is: Based on this motion vector, the system identifies spatially adjacent data nodes within a preset range outside the visible field of view. The specific identification logic is as follows: taking the current view frustum as the center, the near and far clipping planes and field of view of the view frustum are extended outward by a preset buffer scaling factor to construct an expanded prefetch bounding volume. The spatial indexing model is used to retrieve all data nodes that intersect with the prefetch bounding volume but have not yet been loaded into the video memory, which are used as a set of candidate nodes to be analyzed.

[0105] For each candidate node and each node currently residing in GPU memory, the system maintains a global access history list. This list uses a hash mapping structure, with the unique identifier ID of the data node as the key and a timestamp queue with a capacity of K as the value. Whenever a node is called by the rendering engine or is prefetched, the system records the current system timestamp and pushes it to the head of the corresponding queue. If the queue length exceeds K, the oldest timestamp is removed. The system calculates the cache priority weight of each node based on the LRU-K algorithm combined with motion prediction. The input parameters are defined as follows: viewpoint motion vector. Geometric center coordinates of candidate nodes Current timestamp And the timestamp of the Kth most recent visit in the node's access history list. First, the cosine of the angle between the node and the direction of motion of the viewpoint is calculated as the direction factor. In the calculation, let the vector pointing to the node be... The calculation formula is as follows:

[0106] ;

[0107] Predicted access probability based on direction factor Normalize it to the interval between 0 and 1, and the calculation formula is as follows: .

[0108] Next, calculate the time urgency factor of LRU-K. If a node has fewer than K historical access records, then set =0;

[0109] Otherwise, the calculation formula is:

[0110] ;

[0111] in The preset time normalization constant, To prevent the minimum value where the denominator is zero.

[0112] Final cache priority weight It is derived by weighting the predicted probability with time urgency. The preset weight adjustment coefficient has a value range of 0 to 1, and the calculation formula is as follows:

[0113] ;

[0114] The system iterates through the cache priority weights of all candidate nodes. Set a preset loading threshold When a candidate node Value greater than When the data is in the queue, the system adds it to the pre-buffered rendering data queue. The system then starts an independent IO thread pool to asynchronously read the texture and model data of the data nodes in the queue from disk or network storage, and calls the graphics API to upload it to the video memory.

[0115] At the same time, the system starts a video memory status monitoring daemon to poll the GPU video memory usage in real time and set a preset usage threshold. For example, when the current video memory usage exceeds 80% of the total video memory capacity, it indicates that the video memory usage has reached 80% of the total capacity. When this occurs, the system triggers the cache eviction mechanism. The system iterates through all non-visual data nodes currently residing in video memory and updates their cache priority weights in real time according to the formula mentioned above. and in accordance with The values ​​are sorted from smallest to largest. The system selects the node with the lowest weight as the elimination target, calls the video memory release interface to release its resource occupation, and removes the corresponding record from the access history list until the video memory usage rate falls below the safe threshold.

[0116] By combining the prediction of user motion intent with LRU-K historical access frequency analysis, intelligent video memory resource scheduling is achieved. This ensures that the area the user is about to view can be preloaded to eliminate visual stuttering, and also ensures that frequently accessed hot data can reside in video memory, significantly improving the smoothness of roaming and resource utilization efficiency in large-scale multimodal data scenarios.

[0117] S4. Using the depth buffer and hybrid rendering pipeline, perform layered blending rendering on the pre-buffered rendering data queue to generate an initial hybrid scene view containing two-dimensional layers and three-dimensional models.

[0118] Furthermore, in S4, the layered blending and rendering of the pre-buffered rendering data queue using the depth buffer and the blending rendering pipeline specifically includes the following steps:

[0119] Initialize the blending pipeline, configure the depth test function and color blending mode, and reset the color buffer and depth buffer;

[0120] Based on the rendering layer priority, the pre-buffered rendering data queue is divided into background terrain layer, vector feature layer and 3D model layer;

[0121] Draw the background terrain layer, write the depth values ​​of the corresponding fragments into the depth buffer, and build the depth benchmark of the basic scene;

[0122] The vector feature layer and the 3D model layer are drawn sequentially. The depth value of the fragment to be drawn is compared with the reference value in the depth buffer. Color mixing calculation is performed only on the fragments that pass the test.

[0123] After completing the fragment shading and blending of all levels of data, output the initial blended scene view after compositing.

[0124] Specifically, after completing the pre-scheduled caching of data, the system enters the core rendering and compositing stage. The system first initializes the GPU's blending pipeline, and enables the depth test and color blending functions by calling the graphics API interface. The system configures the depth test comparison function to be less than or equal to, that is, when the depth value of the pixel to be drawn is less than or equal to the value in the current depth buffer, the test is allowed to pass. The color blending mode is configured as source alpha blending mode, that is, the source factor takes the alpha value of the source color, and the target factor takes 1 minus the alpha value of the source color. Subsequently, the system executes a clear command, setting all RGBA channels of the color buffer to 0, i.e., black and transparent, and resetting each pixel bit of the depth buffer to the maximum depth value of 1.0, so as to complete the initialization state reset of the frame buffer object.

[0125] The system reads the pre-buffered rendering data queue in memory, traverses the metadata tags of each data node in the queue, and performs logical layering based on the preset rendering layer priority values. The system divides image tiles and terrain elevation data with priority values ​​of 0 to 10 into the background terrain layer, vector data such as roads, water systems, and administrative divisions with priority values ​​of 11 to 50 into the vector feature layer, and building models and facility equipment models with priority values ​​greater than 50 into the 3D model layer. The system constructs three independent rendering instruction lists and submits them to the rendering pipeline in strict order of background terrain layer, vector feature layer, and 3D model layer.

[0126] The system first renders the background terrain layer. The vertex shader calculates the clipping space coordinates of the vertices based on the terrain elevation data. The rasterizer interpolates to generate fragments. During the fragment operation phase, the system forcibly enables depth write masking for any pixel position in the screen coordinate system. The system calculates the depth value of the current background terrain patch. And write it directly to the corresponding position in the depth buffer. In the process, the color values ​​obtained from texture sampling are written to the color buffer. This process does not involve blending and aims to construct the basic geometric contours and depth reference planes of the entire 3D scene.

[0127] Subsequently, the system sequentially renders the vector feature layer and the 3D model layer. Whether the system disables or keeps the depth write mask enabled depends on the specific semi-transparency requirements, but depth testing must be enabled. For each fragment to be rendered in these two layers, the system executes a depth comparison algorithm, defining the input parameter as the depth value of the current fragment. Compared with the reference depth value already stored in the depth buffer The system executes the following logical judgment formula, where The small offset set to avoid depth conflicts is typically a non-zero positive number for ground-hugging vector features and 0 for ordinary 3D models.

[0128] ;

[0129] Only when the above formula is calculated When true, the fragment passes the test and enters the color blending stage. The system executes the standard Alpha blending algorithm to calculate the final pixel color. The input parameters are defined as: the source fragment color, i.e., the color component of the current layer to be drawn. Source transparency is The target color is the background color component that already exists in the color buffer. Target transparency is The output result is the final color component after mixing. The calculation formula is as follows:

[0130] ;

[0131] ;

[0132] ;

[0133] The system will calculate Update to the color buffer to complete the color fusion for that pixel.

[0134] After all rendering instructions at all levels have been executed, the system calls the buffer swap instruction to submit the rendered background buffer to the foreground display, generating an initial mixed scene view containing two-dimensional layers and three-dimensional models. This step effectively solves the problem of chaotic occlusion relationship between two-dimensional vector data and three-dimensional terrain models when they are superimposed in space through refined depth control and blending calculation, and realizes the correct fusion and semi-transparent superposition display of multi-source data in the same three-dimensional scene.

[0135] S5. Responding to the interaction command for the initial mixed scene view, the preset parameter mapping rules are invoked to convert the two-dimensional vector map elements into three-dimensional rendering data objects and inject them into the rendering pipeline to obtain the updated multi-source fusion visualization results.

[0136] Furthermore, in S5, the process of converting 2D vector map features into 3D rendering data objects and injecting them into the rendering pipeline by calling preset parameter mapping rules specifically includes the following steps:

[0137] Listen for user interaction events in response to the initial mixed scene view, and parse the interaction commands to identify the selected 2D vector map features and their attribute data;

[0138] Based on the geometric type and attribute characteristics of the two-dimensional vector map elements, the corresponding parameter mapping rules are retrieved from the preset style library. The rules define the conversion logic from two-dimensional planar coordinates to three-dimensional spatial coordinates and the appearance rendering parameters.

[0139] The parameter mapping rules are executed to reconstruct the planar geometric data of the two-dimensional vector map features into a three-dimensional geometric mesh, and the attribute data is mapped into three-dimensional rendering attributes to generate a three-dimensional rendering data object.

[0140] Dynamically attach 3D rendering data objects to the list of nodes to be rendered in the hybrid rendering pipeline, triggering scene redraw and refresh, and outputting multi-source fusion visualization results.

[0141] Specifically, after generating the initial blended scene view, the system enters the interactive response phase. It captures user actions on the visual interface through an event listener mechanism. The system uses the input / output manager to register event callback functions for mouse clicks, hovers, or touch gestures. When the user triggers an interactive command at a specific location in the screen coordinate system, the system first obtains the screen pixel coordinates. and and the pixel width of the current rendering window. With height The system converts screen pixel coordinates into normalized device coordinates. and The coordinate range is from -1 to 1, and the calculation formula is as follows:

[0142] ;

[0143] ;

[0144] The ray-picking algorithm is then executed to identify target 2D vector map features, and the system obtains the current camera's observation matrix. With projection matrix Calculate the inverse of the view projection matrix. The calculation formula is: The system constructs a spatial ray emanating from the camera position, defining the direction vector of the ray as... Homogeneous coordinate vectors are introduced in the calculation. Represents a point on the near-cut surface. For points on the far clipping plane, the system first calculates the world coordinates of the near points. and the distant world coordinates :

[0145] ;

[0146] ;

[0147] When performing perspective division and of After component normalization, calculate the normalized ray direction vector. :

[0148] ;

[0149] The system uses the ray to traverse the spatial index of the vector feature layer, calculates the intersection test between the ray and the two-dimensional vector polygon in the scene. If an intersection is detected, the system obtains the unique identifier ID of the vector feature and reads its associated attribute data table from the memory database. This table contains semantic information such as feature type, floor height, and building age, which serves as the basic input data for subsequent transformations.

[0150] Based on the identified feature type attributes, the system retrieves the corresponding parameter mapping rule file from a pre-set style library. This rule file is stored in JSON or XML format and defines the mapping logic from two-dimensional attributes to three-dimensional geometric parameters. The system parses the geometric transformation fields and material definition fields in the rules. For example, the rule defines the height parameter of a building. The layer number equals the layer number in the attribute field multiplied by the layer height constant. The appearance texture ID is determined by the building function attributes. The system extracts the two-dimensional planar geometric data of the vector features, i.e., the polygon vertex sequence, defining the polygon's first vertex. The two-dimensional coordinates of each vertex are and The polygon has a total of For each vertex, the system first obtains the terrain elevation value at that location by querying the depth texture or elevation data of the background terrain layer. Then, the stretching algorithm in the parameter mapping rule is executed to reconstruct the two-dimensional point set into a three-dimensional mesh. For each vertex... The system generates the corresponding bottom vertex. With top vertex The calculation formula is as follows:

[0151] ;

[0152] ;

[0153] The system constructs an index sequence for the lateral triangular mesh based on the right-hand screw rule. For connecting the first... The vertex and the first Given edges from vertices, generate two triangular faces with vertex indices of and . and For the top cap, the system uses the ear-cutting method or the Delaunay triangulation algorithm to mesh the top vertex set and generate a set of top triangular facets.

[0154] While completing the geometric reconstruction, the system maps the attribute data to 3D rendering attributes. Based on the material parameters in the mapping rules, the system calculates the texture coordinates of the vertices. and Define the texture horizontal scaling factor as The vertical scaling factor is For the side mesh, the system first calculates the cumulative distance of the bottom perimeter, defining the first... The path length from each point to the starting point is , ,for The calculation formula is as follows:

[0155] ;

[0156] Based on this, the texture coordinates of each vertex are calculated, including the bottom vertex. The texture coordinates are Top vertex The texture coordinates are If the rules specify dynamic shader effects, the system encodes the attribute values ​​as floating-point numbers and writes them into a custom channel of the vertex attribute buffer. The system encapsulates the generated vertex position data, normal data, texture coordinate data, and color data into a 3D rendering data object, which is a mesh instance containing a vertex buffer object (VBO) and an index buffer object (IBO).

[0157] Finally, the system calls the scene graph management interface of the rendering engine to dynamically attach the newly generated 3D rendering data object to the list of nodes to be rendered in the hybrid rendering pipeline. Usually, it is attached to the child node of the 3D model layer. The system updates the version number of the scene graph or marks the dirty rectangle area, triggering the redraw refresh mechanism of the rendering loop. During the execution of the rendering pipeline in the next frame, the newly added 3D object will participate in the depth test and color blending process, thereby growing the corresponding 3D entity at the original 2D map position.

[0158] Through parametric real-time modeling technology, the dynamic transformation from abstract two-dimensional data to concrete three-dimensional models is realized, enabling users to load and display high-precision three-dimensional information on demand through simple interactive operations. This not only enriches the information dimensions of the visualization scene, but also avoids the huge performance overhead caused by loading the full three-dimensional model.

[0159] Example 2:

[0160] When processing large-scale data, 3D scene rendering technology often relies on static, conventional spatial partitioning structures and passive on-demand loading mechanisms, lacking accurate prediction of the user's high-speed viewpoint movement intentions. This results in frequent screen blanking or noticeable visual stuttering caused by data loading lag during rapid navigation. To address these issues, this invention provides a unified multimodal data processing and visualization system, the structure of which is as follows: Figure 3 As shown. The system includes:

[0161] The data standardization mapping module is used to establish a global spatial coordinate system and a unified time reference system, and to map multi-source heterogeneous data such as satellite imagery, vector maps and 3D models to the 3D geospatial rendering environment to generate standardized spatiotemporal data resources.

[0162] The visible field scene extraction module is used to construct a spatial index model for the standardized spatiotemporal data resources based on the adaptive bounding box tree and scene graph data structure, perform collision detection and view frustum culling, and extract the visible field scene data set of the current viewpoint;

[0163] The pre-scheduled cache module is used to calculate the viewpoint motion vector based on the user interaction trajectory, and to use the LRU-K algorithm to pre-read and schedule the spatially adjacent data nodes of the visible scene data set to generate a pre-buffered rendering data queue.

[0164] The layered blending rendering module is used to perform layered blending rendering on the pre-buffered rendering data queue using a depth buffer and a blending rendering pipeline, generating an initial blended scene view containing two-dimensional layers and three-dimensional models.

[0165] The dynamic interaction synchronization module is used to respond to the interaction commands for the initial mixed scene view, call the preset parameter mapping rules to convert the two-dimensional vector map elements into three-dimensional rendering data objects and inject them into the rendering pipeline to obtain the updated multi-source fusion visualization results.

[0166] Specifically, the data standardization mapping module is configured as a data preprocessing unit running on the central processing unit. It uses a built-in projection transformation library or matrix operation logic to uniformly convert the raw data from different geographic coordinate systems into Cartesian spatial coordinates in the geocentric rectangular coordinate system or the projected coordinate system. It also uniformly maps the time information of multi-source data to the standard time axis and stores it in the standardized spatiotemporal database.

[0167] The visible scene extraction module maintains an adaptive bounding box tree or octree spatial index structure in memory that is synchronized with the rendering scene. It constructs the view frustum geometry in real time and uses the separating axis theorem to quickly calculate the intersection state of the scene graph node bounding box and the view frustum to eliminate invalid rendering nodes.

[0168] The pre-scheduled cache module acts as an input / output scheduling controller between video memory and main memory. It calculates the viewpoint motion vector and predicts the future visible area to initiate pre-fetch requests to the background. At the same time, it maintains a video memory resource replacement list based on the least recently used K algorithm to manage data residency.

[0169] The layered blending rendering module is based on the programmable rendering pipeline of the graphics processing unit, integrates a depth buffer and a color mixer, strictly organizes the drawing queue according to the priority order of the background terrain layer, vector feature layer and 3D model layer, sets depth writing and testing strategies for different layers, and uses alpha blending technology to process layer overlay to synthesize a blended image without depth conflicts.

[0170] The dynamic interaction synchronization module, as a combination of input event listener and parametric modeling engine, captures user operations in real time and retrieves the identifier of the selected two-dimensional vector feature through ray casting algorithm. It calls the preset parameter mapping rules to parse attribute data, uses shaders to stretch and triangulate the two-dimensional plane vertices into three-dimensional mesh objects, and finally dynamically attaches the generated vertex buffer objects to the rendering nodes of the scene graph to achieve real-time visualization updates.

[0171] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for unified processing and visualization of multimodal data, characterized in that, Includes the following steps: S1. Establish a global spatial coordinate system and a unified time reference system to map multi-source heterogeneous data such as satellite imagery, vector maps and 3D models to the 3D geospatial rendering environment and generate standardized spatiotemporal data resources. S2. Based on the adaptive bounding box tree and scene graph data structure, construct a spatial index model for the standardized spatiotemporal data resources, perform collision detection and view frustum culling, and extract the set of scene data in the visible field of view from the current perspective. S3. Calculate the viewpoint motion vector based on the user interaction trajectory, and use the LRU-K algorithm to pre-read and schedule the spatially adjacent data nodes of the visible scene data set to generate a pre-buffered rendering data queue. S4. Using a depth buffer and a hybrid rendering pipeline, perform layered fusion rendering on the pre-buffered rendering data queue to generate an initial hybrid scene view containing two-dimensional layers and three-dimensional models. S5. In response to the interaction command for the initial mixed scene view, call the preset parameter mapping rules to convert the two-dimensional vector map elements into three-dimensional rendering data objects and inject them into the rendering pipeline to obtain the updated multi-source fusion visualization result.

2. The method for unified processing and visualization of multimodal data according to claim 1, characterized in that, In step S1, generating standardized spatiotemporal data resources specifically includes the following steps: The CGCS2000 coordinate system is established as the global spatial coordinate system, and the unified time reference system is established. The raster data of satellite imagery, the two-dimensional vector map elements of vector maps, and the grid data of three-dimensional models are acquired respectively. Perform coordinate projection transformation and timestamp alignment on the raster data, the two-dimensional vector map elements, and the grid data to uniformly map them to the global spatial coordinate system and the unified time reference system; Using a matching algorithm based on timestamps and spatial indexes, the aligned data is mounted to the corresponding level node of the three-dimensional geospatial rendering environment to obtain the standardized spatiotemporal data resources.

3. The method for unified processing and visualization of multimodal data according to claim 1, characterized in that, In step S2, a spatial indexing model is specifically constructed for the standardized spatiotemporal data resources based on the adaptive bounding box tree and scene graph data structure. Includes the following steps: The various entity objects in the standardized spatiotemporal data resources are analyzed, and a scene graph data structure containing root nodes, transformation matrix nodes and leaf geometry nodes is constructed based on their entity attributes and logical relationships. Calculate the bounding box of each leaf geometric node in the scene graph data structure, and generate the corresponding adaptive bounding box tree using a bottom-up aggregation strategy; Establish bidirectional mapping pointers between the spatial nodes of the adaptive bounding box tree and the logical nodes of the scene graph data structure to generate a spatial index model that associates spatial locations with logical attributes.

4. The method for unified processing and visualization of multimodal data according to claim 1, characterized in that, In step S2, the collision detection and frustum removal process specifically includes the following steps: Obtain the camera parameters and projection matrix of the current viewpoint, and construct the view frustum space containing six clipping planes; Traverse the spatial index model from the root node and perform an intersection test between the node bounding boxes of the adaptive bounding box tree and the spatial extent of the view frustum; Node branches that are disjoint from the spatial range of the view frustum are excluded, nodes that intersect or contain each other are retained, and geometric information is extracted from the corresponding scene graph data structure to generate a set of viewable scene data.

5. The method for unified processing and visualization of multimodal data according to claim 1, characterized in that, In step S3, the pre-fetching and memory scheduling of spatially adjacent data nodes in the visible scene data set using the LRU-K algorithm specifically includes the following steps: Record the historical access timestamps of each data node in the visible field scene data set, and maintain an access history list containing K historical access records; The predicted access probability of spatially adjacent data nodes within a preset range outside the visible field boundary is calculated based on the viewpoint motion vector of the current viewpoint, and the cache priority weight of each data node is calculated in combination with the access history list. The adjacent data nodes with cache priority weights higher than the preset loading threshold are asynchronously loaded into video memory and added to the pre-buffered rendering data queue. Monitor the video memory usage status. When the usage rate exceeds the preset usage threshold, retrieve and release the video memory space occupied by data nodes that meet the eviction conditions according to the LRU-K rule.

6. The method for unified processing and visualization of multimodal data according to claim 1, characterized in that, In step S4, the layered fusion rendering of the pre-buffered rendering data queue using a depth buffer and a hybrid rendering pipeline specifically includes the following steps: Initialize the blending pipeline, configure the depth test function and color blending mode, and reset the color buffer and depth buffer; Based on the rendering layer priority, the pre-buffered rendering data queue is divided into a background terrain layer, a vector feature layer, and a 3D model layer; The background terrain layer is drawn, and the depth values ​​of the corresponding fragments are written into the depth buffer to construct the depth reference of the basic scene. The vector feature layer and the 3D model layer are drawn sequentially. The depth value of the fragment to be drawn is compared with the reference value in the depth buffer. Color mixing calculation is performed only on the fragments that pass the test. After completing the fragment shading and blending of all levels of data, output the initial blended scene view after compositing.

7. The method for unified processing and visualization of multimodal data according to claim 1, characterized in that, In step S5, the process of converting two-dimensional vector map features into three-dimensional rendering data objects and injecting them into the rendering pipeline by calling preset parameter mapping rules specifically includes the following steps: Listen for user operation events in response to the initial mixed scene view, and parse the interaction instructions to identify the selected two-dimensional vector map features and their attribute data; Based on the geometric type and attribute characteristics of the two-dimensional vector map elements, the corresponding parameter mapping rules are retrieved from the preset style library. The rules define the conversion logic from two-dimensional planar coordinates to three-dimensional spatial coordinates and the appearance rendering parameters. By executing the parameter mapping rules, the planar geometric data of the two-dimensional vector map features are reconstructed into a three-dimensional geometric mesh, and the attribute data is mapped into three-dimensional rendering attributes to generate a three-dimensional rendering data object. The 3D rendering data object is dynamically attached to the list of nodes to be rendered in the hybrid rendering pipeline, triggering a scene redraw and refresh, and outputting a multi-source fusion visualization result.

8. A unified processing and visualization system for multimodal data, characterized in that, A method for unified processing and visualization of multimodal data as described in any one of claims 1-7, the system comprising: The data standardization mapping module is used to establish a global spatial coordinate system and a unified time reference system, and to map multi-source heterogeneous data such as satellite imagery, vector maps and 3D models to the 3D geospatial rendering environment to generate standardized spatiotemporal data resources. The visible field scene extraction module is used to construct a spatial index model for the standardized spatiotemporal data resources based on the adaptive bounding box tree and scene graph data structure, perform collision detection and view frustum culling, and extract the visible field scene data set of the current viewpoint; The pre-scheduled cache module is used to calculate the viewpoint motion vector based on the user interaction trajectory, and to use the LRU-K algorithm to pre-read and schedule the spatially adjacent data nodes of the visible scene data set to generate a pre-buffered rendering data queue. The layered blending rendering module is used to perform layered blending rendering on the pre-buffered rendering data queue using a depth buffer and a blending rendering pipeline, generating an initial blended scene view containing two-dimensional layers and three-dimensional models. The dynamic interaction synchronization module is used to respond to the interaction commands for the initial mixed scene view, call the preset parameter mapping rules to convert the two-dimensional vector map elements into three-dimensional rendering data objects and inject them into the rendering pipeline to obtain the updated multi-source fusion visualization results.

Images