City-level CIM scene streaming rendering method based on multi-modal fusion

Abstract

The invention discloses a city-level CIM scene streaming rendering method based on multi-modal fusion, and the method comprises the following steps: S1, collecting city data, and carrying out the preprocessing of the city data; s2, establishing a cross-modal entity alignment relationship, and constructing a city object structure body; s3, dividing a space by adopting a quadtree method, generating rendering sub-blocks and distributing a unique space code; s4, binding the rendering sub-blocks with the urban object structure body, constructing a multi-modal fusion network, and generating a multi-modal rendering unit set; s5, constructing an inter-frame rendering cost function, and scheduling and sequencing the multi-modal rendering unit set to form an inter-frame scheduling sequence; s6, loading the rendering units according to the inter-frame scheduling sequence to form a city-level three-dimensional view of continuous frames; and S7, updating the scheduling sequence, dynamically adjusting the loading sequence, and realizing stable and efficient streaming rendering of the city-level CIM scene. According to the method, the fusion efficiency and rendering performance of the urban large-scale three-dimensional data are improved, and the method has remarkable engineering applicability and popularization value.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data fusion and urban information modeling, and particularly relates to a method for streaming rendering of urban-level CIM scenes based on multimodal fusion. Background Art

[0002] With the continuous acceleration of the urbanization process, urban information modeling (CIM) technology has gradually become an important supporting means for urban digital management and smart city construction. The CIM system uses a three-dimensional space as a carrier to integrate various types of urban data such as building information models (BIM), geographic information systems (GIS), remote sensing images, Internet of Things sensing data, traffic data, and socio-economic data, and is committed to building a city-level digital twin platform with multi-dimensional attributes such as space, semantics, and time. Through this platform, not only can the three-dimensional visualization of urban infrastructure be realized, but also advanced functions such as multi-source data integration, dynamic monitoring, trend prediction, and emergency response in complex urban scenarios can be supported.

[0003] At present, there have been a large number of studies on visualization rendering technologies for CIM systems. Traditional methods mostly rely on static loading mechanisms to build and present urban three-dimensional models as a whole. However, with the expansion of the urban spatial scale and the access of multi-source data, traditional static rendering methods face many problems such as large storage overhead, slow loading speed, high response latency, and poor interaction performance. In addition, the modeling and visualization methods of single-modal data can no longer meet the needs of urban managers for the comprehensive expression of multi-dimensional attributes of urban objects, and cannot realize the unified presentation and linkage display of information such as building geometric forms, spatial positions, functional semantics, and time states.

[0004] For the rendering optimization of large-scale three-dimensional urban models, some studies have proposed methods based on level of detail (LOD), region clipping, graphics buffer optimization, etc. to reduce the load of single-frame rendering. However, these methods usually focus on the graphics processing level and lack a comprehensive modeling of urban object semantics, time dimensions, and multimodal data structures. Their rendering objects are often static three-dimensional mesh data, which fails to effectively support the update and drive of dynamic information and is difficult to meet the requirements of real-time and data linkage during urban operation.

[0005] In terms of data fusion, in existing CIM systems, the fusion of multi-modal data generally relies on manual annotation or static rule matching, lacking a unified data structure and automated alignment mechanism. Due to problems such as dimensional differences, inconsistent spatial resolutions, and non-uniform time granularities among source data such as building information, geographic information, and remote sensing images, current multi-modal fusion methods can often only achieve local linkage and it is difficult to construct a globally consistent urban object model. In addition, semantic information mostly exists in the form of additional attributes in existing models and fails to be integrated into the graph structure expression between urban objects, resulting in difficulties in performing semantic-driven efficient scheduling and rendering in three-dimensional space.

[0006] In addition, in terms of streaming rendering, current urban-level visualization technologies mainly focus on offline rendering or full-load loading. Even if some studies attempt to use incremental loading or sub-region rendering methods, their inter-frame scheduling strategies are usually based on distance priority or static priority sorting, lacking comprehensive consideration of real-time data changes and semantic correlation degrees. This method causes the system to respond slowly when processing high-frequency updated data, and there are obvious defects in the visual continuity and semantic consistency of the rendering results.

[0007] In the prior art, there are also few methods that use the semantic labels of urban objects as the basis for the association between graph structure nodes to construct a semantic graph structure that can be used to drive scheduling and enhance the expression ability. The semantic graph not only helps to improve the organizational structure of the model but also provides a loading order and content focusing ability that is more in line with the scene logic during rendering. However, there is currently a lack of an efficient expression and scheduling mechanism that simultaneously integrates spatial, semantic, and temporal dimensions, making multi-modal CIM data often only presented one-sidedly and isolated from each other during the streaming rendering process and it is difficult to achieve systematic display and response.

[0008] Furthermore, in terms of the spatial division of three-dimensional urban modeling, although there are already spatial division algorithms such as quadtrees and octrees used to construct the rendering index structure, existing applications mostly focus on frustum culling and graphics simplification processing, failing to be effectively bound to the semantic and temporal dimensions of urban objects and also failing to introduce cross-modal data mapping logic during the division process, resulting in the rendering sub-blocks lacking semantic expression ability and being unable to support the visualization requirements based on semantic aggregation.

[0009] Therefore, how to provide a method for streaming rendering of urban-level CIM scenes based on multi-modal fusion is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0010] An object of the present invention is to propose a method for streaming rendering of urban-level CIM scenes based on multimodal fusion. The present invention makes full use of strategies such as spatial partitioning, semantic graph structure construction, and inter-frame scheduling optimization, and details the construction method of fusing geometric features, semantic labels, and timestamp information in the urban object structure. The quadtree is used for spatial partitioning, a semantic association graph network is constructed, and the streaming scheduling and real-time update of multimodal rendering units are realized through an inter-frame rendering cost function, which has the advantages of strong scalability, high rendering efficiency, good semantic consistency, and strong dynamic response ability.

[0011] A method for streaming rendering of urban-level CIM scenes based on multimodal fusion according to an embodiment of the present invention includes the following steps:

[0012] S1. Collect urban data and perform preprocessing;

[0013] S2. Based on the preprocessed urban data, establish cross-modal entity alignment relationships and construct an urban object structure including geometric features, semantic labels, and timestamp information;

[0014] S3. Use the quadtree method to perform spatial partitioning on the geometric features in the urban object structure, map the divided spatial regions into rendering sub-blocks of a fixed size, and assign unique spatial codes;

[0015] S4. Bind the rendering sub-blocks to the urban object structure, construct a multimodal fusion network with a semantic label graph structure, and generate a set of multimodal rendering units with spatial codes and semantic codes;

[0016] S5. Construct an inter-frame rendering cost function in combination with timestamp information, and use a frame-level optimal path search algorithm to schedule and sort the set of multimodal rendering units to form an inter-frame scheduling sequence;

[0017] S6. Before the start of rendering each frame, determine the rendering sub-blocks corresponding to the current frame in combination with the inter-frame scheduling sequence, and load the rendering units with the highest priority to form a continuous-frame urban-level three-dimensional view;

[0018] S7. Based on the urban-level three-dimensional view, update the inter-frame scheduling sequence, dynamically adjust the loading order of the rendering sub-blocks and rendering units of subsequent frames, and realize the stable and efficient streaming rendering of the urban-level CIM scene.

[0019] Optionally, the urban data includes building information data, remote sensing image data, and Internet of Things perception data, and the preprocessing includes spatial projection conversion, timestamp alignment, and semantic label mapping.

[0020] Optionally, the semantic tags are extracted by parsing the classification fields in the remote sensing image data, building information data, and Internet of Things perception data, and a unique mapping relationship is established with the corresponding geometric features and timestamp information to form a semantic description vector for rendering scheduling.

[0021] Optionally, the semantic tag graph structure is a weighted undirected graph. Each node in the graph corresponds to a city object structure body. The edge weight of each edge is jointly calculated by the cosine similarity between semantic tag vectors and the spatial coexistence relationship. The edge weight is mapped using a logical normalization function, enabling the semantic tag graph to have differentiable, clusterable, and propagable embedding representation capabilities, which is used to enhance the connectivity and expression integrity of the multi-modal rendering unit at the semantic level.

[0022] Optionally, the S2 specifically includes:

[0023] S21. Based on the preprocessed building information data, extract three-dimensional geometric structure parameters. The structure parameters include building height, boundary contour, and outer facade grid point set, denoted as the set G = {g1, g2,..., g m}, where g i represents the geometric feature vector of the i-th building, and m is the number of buildings;

[0024] S22. Compare the preprocessed remote sensing image data with the building information data for reference and extract semantic description tags. The tags include building type, functional attribute, and construction status, and construct a semantic tag set S = {s1, s2,..., s m}, where s i represents the semantic tag vector corresponding to the i-th building;

[0025] S23. Based on the timestamp information in the Internet of Things perception data, construct a multi-source time synchronization sequence T = {t1, t2,..., t m}, where t<~ i represents the time node associated with the i-th building;

[0026] S24. Based on the geometric feature vector g i , semantic tag vector s i , and time node t i , with the goal of minimizing the inter-modal difference degree, establish a cross-modal entity alignment mapping relationship;

[0027] S25. Combine g i , s i , and t i to form a city object structure body O i = (g i , s i , t i), and construct a set of urban object structures \(O = \{O_1, O_2, \ldots, O\}\) m for subsequent rendering unit generation and scheduling.

[0028] Optionally, the step S3 specifically includes:

[0029] S31. Extract the geometric feature vector \(g\) from each structure in the set of urban object structures \(O = \{O_1, O_2, \ldots, O\}\) m where the \(g\) i contains a six - tuple of boundary coordinates of the building i where represents the minimum coordinate value of the \(i\) - th object in the \(X\) - axis direction, represents the maximum coordinate value of the \(i\) - th object in the \(X\) - axis direction, represents the minimum coordinate value of the \(i\) - th object in the \(Y\) - axis direction, represents the maximum coordinate value of the \(i\) - th object in the \(Y\) - axis direction, represents the minimum coordinate value of the \(i\) - th object in the \(Z\) - axis direction, represents the maximum coordinate value of the \(i\) - th object in the \(Z\) - axis direction, and \(m\) is the number of buildings;

[0030] S32. Construct a two - dimensional geographic projection plane, and establish a global rendering bounding box \(B\) with the horizontal projection center of the building as the reference point to form the root - node area:

[0031] \(B=[X\) min , \(X\) max , \(Y\) min , \(Y\) max ;

[0032]

[0033] Traverse \(i = 1, 2, \ldots, m\), where \(max\) represents the maximum - value function and \(min\) represents the minimum - value function;

[0034] S33. Set the minimum size of the rendering sub - block as \(d_0\), and terminate the partitioning process when the width or height of any sub - area is less than \(d_0\) to generate a set of rendering sub - blocks \(Q=\{q_1, q_2, \ldots, q\}\) k where \(q\) j represents the \(j\) - th leaf - node rendering sub - block, and \(k\) is the number of rendering sub - blocks;

[0035] S34. Assign a unique spatial code \(C\) j to each rendering sub - block \(q\) j where the spatial code \(C\) j is constructed according to the depth - first order and the node - path generation rule:

[0036]

[0037] Among them, C j represents the spatial encoding of the j-th rendering sub-block, D represents the depth of the rendering sub-block in the quadtree, and I n ∈{0, 1, 2, 3} represents the quadrant index of the n-th layer in the path from the root node to the j-th leaf node. The quadrant index is numbered from 0 for the first quadrant to 3 for the third quadrant in the clockwise direction;

[0038] S35. Project the geometric feature g i in each urban object structure O i onto the geographic plane, and determine whether the boundary coordinates fall into the spatial area covered by the rendering sub-block q j . If the spatial inclusion relationship is satisfied, bind the urban object structure O i to the corresponding rendering sub-block q j and record the spatial encoding.

[0039] Optionally, the root node area is constructed through the bounding box B and recursively divided using the quadtree partitioning strategy:

[0040]

[0041] Among them, represents the spatial range of the j-th area in the (n + 1)-th layer, are the boundary coordinate values of the j-th area, that is, the east-west boundary and the north-south boundary coordinates.

[0042] Optionally, the specific steps of S4 are as follows:

[0043] S41. Based on the rendering sub-block set Q = {q1, q2,..., q k} and its spatial encoding set C = {C1, C2,..., C k}, assign each structure O m in the urban object structure set O = {O1, O2,..., O i} = (g i , s i , t i ) to the corresponding rendering sub-block q j , and record the binding relationship pair (O i , C j ), where k is the number of rendering sub-blocks and m is the number of buildings;

[0044] S42. Construct the semantic label adjacency matrix A m based on the semantic label set S = {s1, s2,..., s s}, and calculate the semantic adjacency weights between any two urban objects using the similarity function:

[0045]

[0046] Among them, A s (i, j) represents the urban object structure O i and O j adjacency weight in the semantic dimension, cos(s i , s j ) represents the cosine similarity between the semantic label vectors s i and s j , δ ij = 1 indicates that the urban object structure O i and O j are in the same rendering sub-block, δ ij = 0 indicates that they are in different rendering sub-blocks, α and β are semantic similarity weight parameters, and exp(·) is the natural exponential function with base e;

[0047] S43. Construct a multimodal fusion graph structure G f = (V, E), where the node set V = {v1, v2,..., v m}, and each node v i represents the urban object structure O i , and the edge set E includes two types: geometric adjacency edges and semantic label edges. The fusion edge weight calculation formula is as follows:

[0048]

[0049] Among them, W(i, j) represents the edge weight between nodes v i and v j in the fusion graph, ‖g i - g j ‖2 represents the Euclidean distance between geometric feature vectors, ‖g i ‖2 represents the two-norm of the geometric feature vector g i , A s (i, j) is the corresponding element in the semantic adjacency matrix, and λ1, λ2 are two types of edge weight distribution coefficients, and max represents the maximum value function;

[0050] S44. Based on the multimodal fusion graph structure and the bound spatial encoding, generate a multimodal rendering unit set U = {u1, u2,..., u k}, where each rendering unit u j contains the spatial encoding C j of the rendering sub-block, the index set of the affiliated urban object structure and the corresponding fusion graph sub-structure, which are used for subsequent inter-frame scheduling and graphics rendering.

[0051] Optionally, S5 specifically includes:

[0052] S51. Extract the index set I of the city object structure corresponding to each rendering unit from the multi-modal rendering unit set, j , and obtain the corresponding multi-source time synchronization sequence T = {t1, t2,..., t m}, where each rendering unit u j includes the spatial encoding C of the rendering sub-block j , t i represents the time node associated with the i-th building, k is the number of rendering sub-blocks, and m is the number of buildings;

[0053] S52. Construct an inter-frame rendering cost function to measure the switching overhead between any two rendering units u i and u j in consecutive frames:

[0054]

[0055] where F(i, j) represents the inter-frame cost from rendering unit u i to u j , |I i ∩ I j | represents the number of shared objects in the two units, t p is the timestamp of the city object structure O p , τ is the current frame reference time, s p is the semantic label vector, is the average value of the semantic label vectors contained in the rendering unit u j , η1, η2, and η3 are the weighting coefficients of the structural difference, time difference, and semantic difference, and exp(·) is the natural exponential function with base e;

[0056] S53. Construct a frame-level optimal path search model, with the goal of minimizing the total rendering cost of the loading order of all rendering units within consecutive frames:

[0057]

[0058] where min represents the minimum value function, π = {π1, π2,..., π k} represents the access order of the rendering units from the first frame to the k-th frame, k is the number of rendering sub-blocks, Π represents the set of all access paths, F(π n , π n+1 ) represents the rendering cost between two consecutive frames, I πn represents the object index set included in the rendering unit of the n-th frame, and μ is the path smoothing control factor. The path similarity term is used to penalize too large a change in the object set between consecutive frames;

[0059] S54. Obtain the optimal scheduling path π according to the search results * , and use the rendering unit sequence corresponding to each frame in the path as the inter-frame scheduling sequence to control the loading order and time distribution of the multi-modal rendering units.

[0060] Optionally, the S6 specifically includes:

[0061] S61. Based on the inter-frame scheduling sequence π * ={π1, π2,..., π k}, determine the multi-modal rendering units to be loaded when the current frame index is n and obtain the rendering sub-block code bound to the rendering unit and the set of urban object structure indices

[0062] S62. Based on the multi-modal fusion graph structure G f , extract the set of sub-graph nodes included in the rendering unit and sort the nodes according to the priority function: where V represents the set of nodes, P(i) represents the rendering priority value of the urban object structure O

[0063]

[0064] in the current frame, t i represents the time node associated with the i-th building, τ i is the reference time of the current frame, n is the set of adjacent nodes of the urban object structure O in the graph structure G i , f is the set of object indices loaded in the previous frame, s is the semantic label vector of the urban object structure O i i i i is the mean of all semantic vectors of the current rendering unit, γ, θ1, θ2, and θ3 are the time change rate adjustment coefficient, time-driven weight, adjacent difference weight, and semantic difference weight respectively, and exp(·) is the natural exponential function with e as the base;

[0065] S63. Sort all objects in according to the priority function, and load the urban object structure and the corresponding bound data in descending order to construct the 3D view content required for the current frame;

[0066] S64. Locate and render sub-blocks based on the spatial encoding of the loaded object, organize the spatial arrangement order of all rendered sub-blocks on the two-dimensional geographic projection plane, and merge them to form a complete city-level three-dimensional view of the current frame.

[0067] The beneficial effects of the present invention are as follows:

[0068] First, by constructing a city object structure that integrates geometric features, semantic tags, and timestamp information, the present invention realizes the unified modeling and efficient expression of multi-modal city data, overcoming the problems of inconsistent multi-source data formats, non-uniform granularity, and difficult spatio-temporal alignment in the prior art. By establishing cross-modal entity alignment relationships, BIM, remote sensing images, and Internet of Things perception data are organically integrated, effectively improving the data consistency and expression integrity of the city information model, and laying a foundation for subsequent semantic analysis and rendering scheduling.

[0069] Second, in terms of spatial organization, the present invention uses the quadtree method to hierarchically divide the city's three-dimensional space, maps the city area into rendering sub-blocks of a fixed size, and assigns a unique spatial encoding to each sub-block, realizing the orderly division and rapid indexing of large-scale city spaces. At the same time, by constructing a semantic tag graph structure and a multi-modal fusion network, the spatial division is deeply bound to semantic associations, enabling each rendered sub-block to have a schedulable semantic expression ability, thus significantly improving the accuracy of content aggregation and visual focus positioning in the rendering scheduling process.

[0070] Finally, the present invention constructs an inter-frame rendering cost function based on timestamp information, designs a frame-level optimal path search algorithm to realize the dynamic scheduling and sorting of multi-modal rendering units, and combines the current frame state with user interaction behaviors to achieve incremental loading and semantic-driven rendering between consecutive frames in the city-level CIM scenario. By introducing a dynamic scheduling mechanism and a real-time adjustment strategy, the problems of full-volume loading, response delay, and rendering jamming in the prior art are effectively solved, ensuring the stability and high-performance performance of the system under high-frequency updates, multi-source inputs, and complex interaction conditions, and realizing real-time, smooth, and efficient rendering of ultra-large-scale city three-dimensional scenes. Description of the Drawings

[0071] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, and do not constitute a limitation to the present invention. In the drawings:

[0072] Figure 1 is a flowchart of a method for streaming rendering of a city-level CIM scenario based on multi-modal fusion proposed by the present invention;

[0073] Figure 2Flowchart for constructing the urban object structure of a city-level CIM scene streaming rendering method based on multi-modal fusion proposed by the present invention;

[0074] Figure 3 Flowchart for spatial partitioning and binding of geometric features of the urban object structure of a city-level CIM scene streaming rendering method based on multi-modal fusion proposed by the present invention. Detailed implementation manners

[0075] Now, the present invention will be further described in detail with reference to the accompanying drawings. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic manner, so they only show the components related to the present invention.

[0076] Reference Figures 1-3 , a city-level CIM scene streaming rendering method based on multi-modal fusion, includes the following steps:

[0077] S1. Collect urban data and perform preprocessing;

[0078] S2. Based on the preprocessed urban data, establish cross-modal entity alignment relationships, and construct an urban object structure including geometric features, semantic labels, and timestamp information;

[0079] S3. Use the quadtree method to perform spatial partitioning on the geometric features in the urban object structure, map the partitioned spatial regions to rendering sub-blocks of a fixed size, and assign unique spatial codes;

[0080] S4. Bind the rendering sub-blocks to the urban object structure, construct a multi-modal fusion network with a semantic label graph structure, and generate a set of multi-modal rendering units with spatial codes and semantic codes;

[0081] S5. Construct an inter-frame rendering cost function in combination with timestamp information, and use a frame-level optimal path search algorithm to schedule and sort the set of multi-modal rendering units to form an inter-frame scheduling sequence;

[0082] S6. Before the start of each frame rendering, determine the rendering sub-blocks corresponding to the current frame in combination with the inter-frame scheduling sequence, and load the rendering units with the highest priority to form a city-level three-dimensional view of consecutive frames;

[0083] S7. Based on the city-level three-dimensional view, update the inter-frame scheduling sequence, dynamically adjust the loading order of the rendering sub-blocks and rendering units of subsequent frames, and achieve stable and efficient streaming rendering of the city-level CIM scene.

[0084] This paper proposes a comprehensive streaming rendering method for city-level CIM scenarios based on multimodal fusion, integrating the entire process of urban data collection, structured modeling, spatial organization, semantic graph construction, inter-frame scheduling, and 3D rendering. This approach enables the dynamic organization and streaming visualization of multi-source urban data. Compared to traditional static loading or single-modal rendering solutions, this method offers advantages such as strong semantic integrity, low rendering latency, and fast system response, meeting the dual requirements of real-time and interactivity for large-scale urban digital twin scenarios.

[0085] In this embodiment, the urban data includes building information data, remote sensing image data and Internet of Things perception data, and the preprocessing includes spatial projection conversion, timestamp alignment and semantic label mapping.

[0086] The present invention clearly proposes a preprocessing method for urban data, including spatial projection transformation, timestamp alignment and semantic label mapping, which effectively solves problems such as inconsistent spatial benchmarks, misaligned time dimensions and inconsistent semantic expressions among multi-source heterogeneous data, improves the fusion accuracy of multimodal data and downstream processing efficiency, and provides a solid foundation for building a consistent urban object structure.

[0087] In this embodiment, the semantic tags are extracted by parsing the classification fields in the remote sensing image data, building information data and IoT perception data, and a unique mapping relationship is established with the corresponding geometric features and timestamp information to form a semantic description vector that can be used for rendering scheduling.

[0088] By extracting and unifying semantic labels from remote sensing images, building information, and IoT perception data, and establishing a one-to-one mapping with geometric features and timestamps, the present invention achieves a strong association between multi-dimensional attributes, enabling each urban object to have not only geometric form and spatial location, but also semantic classification and temporal information, thus enhancing the semantic driving capability of rendering scheduling and the depth of scene understanding.

[0089] In this embodiment, the semantic label graph structure is a weighted undirected graph, in which each node corresponds to a city object structure. The edge weight of each edge is obtained by jointly calculating the cosine similarity and spatial coexistence relationship between the semantic label vectors. The edge weight is mapped using a logical normalization function, so that the semantic label graph has differentiable, clusterable and propagable embedding representation capabilities, which is used to enhance the connectivity and expression integrity of the multimodal rendering unit at the semantic level.

[0090] The present invention constructs a weighted undirected graph structure using semantic tags, and introduces cosine similarity and spatial coexistence relationship to jointly calculate the edge weights, enabling the semantic tag graph to have embedding ability, propagation ability and structural expression ability, effectively improving the semantic connectivity between objects. This graph structure can be used for semantic aggregation, path optimization and high-frequency area recognition during rendering, enhancing the intelligence of the system in rendering organization and scheduling.

[0091] In this embodiment, the S2 specifically includes:

[0092] S21. Based on the preprocessed building information data, extract three-dimensional geometric structure parameters, and the structure parameters include building height, boundary contour, and outer facade grid point set, denoted as set G = {g1, g2,..., g m}, where g i represents the geometric feature vector of the i-th building, and m is the number of buildings;

[0093] S22. Compare the preprocessed remote sensing image data with the building information data for reference, extract semantic description tags, and the tags include building type, functional attribute, and construction status, and construct a semantic tag set S = {s1, s2,..., s m}, where s i represents the semantic tag vector corresponding to the i-th building;

[0094] S23. Based on the timestamp information in the Internet of Things perception data, construct a multi-source time synchronization sequence T = {t1, t2,..., t m}, where t i represents the time node associated with the i-th building;

[0095] S24. Based on the geometric feature vector g i , semantic tag vector s i and time node t i , with the goal of minimizing the inter-modal difference degree, establish a cross-modal entity alignment mapping relationship;

[0096] S25. Combine g i , s i and t i to form a city object structure body O i = (g i , s i , t i ), and construct a city object structure body set O = {O1, O2,..., O m} for subsequent rendering unit generation and scheduling.

[0097] The present invention forms a multimodal rendering unit by constructing a unified set of urban object structure bodies, integrating geometric, semantic, and temporal dimension information, and combining semantic adjacency relationships and geometric topological structures, enabling the system to achieve graph-based structured visualization scheduling. Compared with the traditional rendering method based on building IDs or spatial blocks, the multimodal unit has stronger data integrity and flexible rendering organization capabilities.

[0098] In this embodiment, S3 specifically includes:

[0099] S31. Extract the geometric feature vector g from each structure body in the set of urban object structure bodies O = {O1, O2,..., O m}, where the g i contains a six-tuple of the boundary coordinates of the building i where represents the minimum coordinate value of the i-th object in the X-axis direction, represents the maximum coordinate value of the i-th object in the X-axis direction, represents the minimum coordinate value of the i-th object in the Y-axis direction, represents the maximum coordinate value of the i-th object in the Y-axis direction, represents the minimum coordinate value of the i-th object in the Z-axis direction, represents the maximum coordinate value of the i-th object in the Z-axis direction, and m is the number of buildings;

[0100] S32. Construct a two-dimensional geographic projection plane, use the horizontal projection center of the building as a reference point, establish a global rendering bounding box B, and form a root node area:

[0101] B = [X min , X max , Y min , Y max ;

[0102]

[0103] Traverse i = 1, 2,..., m, where max represents the maximum value function and min represents the minimum value function;

[0104] S33. Set the minimum size of the rendering sub-block as d0, and terminate the division process when the width or height of any sub-region is less than d0, generating a set of rendering sub-blocks Q = {q1, q2,..., q k}, where q j represents the j-th leaf node rendering sub-block, and k is the number of rendering sub-blocks;

[0105] S34. Assign a unique spatial code C to each rendering sub-block q j ​j , the spatial encoding C j is constructed according to the depth - first order and the node - path generation rule:

[0106]

[0107] where C j represents the spatial encoding of the j - th rendering sub - block, D represents the depth of the rendering sub - block in the quadtree, and I n ∈{0, \alpha, 2, 3} represents the quadrant index of the n - th layer in the path from the root node to the j - th leaf node. The quadrant index is numbered from 0 for the first quadrant to 3 for the third quadrant in the clockwise direction;

[0108] S35. Project the geometric feature g i in each urban object structure O i onto the geographic plane, and determine whether the boundary coordinates fall into the spatial region covered by the rendering sub - block q j . If the spatial inclusion relationship is satisfied, bind the urban object structure O i to the corresponding rendering sub - block q j and record the spatial encoding.

[0109] By introducing the quadtree space - partitioning mechanism, the present invention realizes the recursive partitioning of the urban three - dimensional space, organizes the complex urban space into rendering sub - blocks of a fixed size, and assigns spatial encodings to them, significantly improving the spatial retrieval efficiency and scheduling and positioning speed during the rendering process, reducing the complexity of large - scale CIM data management, and supporting the fast - loading logic of streaming rendering. [[ID= \theta]]

[0110] In this embodiment, the root - node region is constructed through the bounding box B and recursively partitioned using the quadtree partitioning strategy:

[0111]

[0112] where, represents the spatial range of the j - th region at the n + 1 - th layer, are the boundary coordinate values of the j - th region, that is, the east - west and north - south boundary coordinates.

[0113] By defining the spatial boundary of the root - node region and adopting the standardized recursive quadtree partitioning rule, a clear spatial - boundary expression form is generated for each layer of sub - blocks, ensuring that the partitioning structure is traceable and logically rigorous. This method provides an accurate spatial basis for the subsequent spatial positioning of rendering sub - blocks and the construction of visualization maps.

[0114] In this embodiment, the specific content of S4 includes:

[0115] It should be noted that in the above translation, \(\alpha\) in the original text is likely a misprint or an undefined symbol. If it has a specific meaning, it needs to be corrected according to the actual situation. Also, \(\theta\) in the translation is used to replace the misprinted or undefined symbol in the original text for the sake of translation integrity.S41. Based on the set of rendering sub - blocks \(Q=\{q_1,q_2,\cdots,q\}\) k and its spatial encoding set \(C = \{C_1,C_2,\cdots,C\}\), k allocate each structure \(O\) in the set of urban object structures \(O=\{O_1,O_2,\cdots,O\}\) m to the corresponding rendering sub - block \(q\), i where \(O=(g\), i ,s\), i ,t\), i ) and record the binding relationship pair \((O,C)\), j where \(k\) is the number of rendering sub - blocks and \(m\) is the number of buildings; i ,C\), j )

[0116] S42. Construct the semantic label adjacency matrix \(A\) m based on the set of semantic labels \(S=\{s_1,s_2,\cdots,s\}\), s and calculate the semantic adjacency weights between any two urban objects using a similarity function:

[0117]

[0118] where \(A(i,j)\) represents the adjacency weight between urban object structures \(O\) s [[ID=3�]]and \(O\) in the semantic dimension, \(\cos(s,s)\) represents the cosine similarity between semantic label vectors \(s\) i and \(s\), \(\delta = 1\) indicates that urban object structures \(O\) j and \(O\) are in the same rendering sub - block, \(\delta = 0\) indicates that they are in different rendering sub - blocks, \(\alpha\) and \(\beta\) are semantic similarity weight parameters, and \(\exp(\cdot)\) is the natural exponential function with base \(e\); i ,s\), j ) i and \(s\), j ) ij and \(O\) i and \(O\) j are in the same rendering sub - block, \(\delta\) ij = 0 indicates that they are in different rendering sub - blocks, \(\alpha\) and \(\beta\) are semantic similarity weight parameters, and \(\exp(\cdot)\) is the natural exponential function with base \(e\);

[0119] S43. Construct a multi - modal fusion graph structure \(G=(V,E)\), f where the node set \(V=\{v_1,v_2,\cdots,v\}\), m and each node \(v\) i represents an urban object structure \(O\), i and the edge set \(E\) contains two types of edges: geometric adjacency edges and semantic label edges. The formula for fusing edge weights is as follows:

[0120]

[0121] where \(W(i,j)\) represents the weight of the edge between nodes \(v\) i and \(v\) jThe edge weight between them, ‖g i -g j ‖2 represents the Euclidean distance between geometric feature vectors, ‖g i ‖2 represents the second norm of the geometric feature vector g i , A s (i,j) is the corresponding element in the semantic adjacency matrix, λ1, λ2 are the two types of edge weight distribution coefficients, and max represents the maximum value function;

[0122] S44. Based on the multi-modal fusion graph structure and bound spatial encoding, generate a multi-modal rendering unit set U = {u1, u2,..., u k}, where each rendering unit u j contains the spatial encoding C j of the rendering sub-block, the set of indices of the corresponding urban object structure and the corresponding fusion graph sub-structure, which are used for subsequent inter-frame scheduling and graphics rendering.

[0123] In the present invention, the rendering sub-block is bound to the urban object, and the semantic adjacency matrix and the fusion graph structure are introduced, so that the multi-modal rendering unit has a complete graph topology representation. By accurately calculating the weighted edge weights of the semantic similarity and the spatial adjacency relationship, the correlation modeling inside and between the rendering units is realized, which provides support for the efficient rendering scheduling and semantic visualization based on the graph structure.

[0124] In this embodiment, the S5 specifically includes:

[0125] S51. Extract the set of indices I j of the corresponding urban object structure of each rendering unit from the multi-modal rendering unit set, and obtain the corresponding multi-source time synchronization sequence T = {t1, t2,..., t m}, where each rendering unit u j contains the spatial encoding C j of the rendering sub-block, t i represents the time node associated with the i-th building, k is the number of rendering sub-blocks, and m is the number of buildings;

[0126] S52. Construct an inter-frame rendering cost function for measuring the switching overhead between any two rendering units u i and u j in consecutive frames:

[0127]

[0128] Among them, F(i,j) represents the inter-frame cost from the rendering unit u i to u j , |I i ∩ I j| Represents the number of shared objects in two units, t p Is the timestamp of the urban object structure O p , τ is the reference time of the current frame, s p Is the semantic label vector, Is the rendering unit u j The average value of the semantic label vectors contained in it, η1, η2, η3 are the weighting coefficients of structural difference, time difference and semantic difference, and exp(·) is the natural exponential function with e as the base;

[0129] S53. Build a frame-level optimal path search model, and set the goal to minimize the total rendering cost of the loading order of all rendering units within consecutive frames: [[ID=1%]]

[0130]

[0131] Among them, min represents the minimum value function, π = {π1, π2,..., π k} represents the access order of rendering units from the first frame to the kth frame, k is the number of rendering sub-blocks, Π represents the set of all access paths, and F(π n , π n+1 ) represents the rendering cost between two consecutive frames, Represents the set of object indices contained in the rendering unit of the nth frame, μ is the path smoothing control factor, and the path similarity term is used to penalize the excessive change in the object set between consecutive frames;

[0132] S54. Obtain the optimal scheduling path π * according to the search result, and use the rendering unit order corresponding to each frame in the path as the inter-frame scheduling sequence to control the loading order and time distribution of multi-modal rendering units.

[0133] By constructing the inter-frame rendering cost function and the frame-level optimal path search algorithm, the present invention realizes the loading order planning of rendering units based on temporal logic and content similarity, effectively reducing the switching cost between consecutive frames. The path smoothing mechanism further suppresses the visual jump problem caused by frequent loading, improving the rendering coherence and system stability.

[0134] In this embodiment, the S6 specifically includes:

[0135] S61. Based on the inter-frame scheduling sequence π * = {π1, π2,..., π k}, determine the multi-modal rendering units to be loaded when the current frame index is n and obtain the rendering sub-block code bound to the rendering unit and the set of urban object structure indices

[0136] S62. Extract the rendering units based on the multimodal fusion graph structure G f , and extract the set of sub-graph nodes contained in and sort the nodes according to the priority function:

[0137]

[0138] where V represents the set of nodes, and P(i) represents the rendering priority value of the urban object structure O i in the current frame, t i represents the time node associated with the i-th building, τ n is the reference time of the current frame, is the urban object structure O i in the graph structure G f the set of adjacent nodes, is the set of object indices loaded in the previous frame, s i is the semantic label vector of the urban object structure O i , is the mean of all semantic vectors of the current rendering unit, γ, θ1, θ2, θ3 are the time change rate adjustment coefficient, time-driven weight, adjacent difference weight, and semantic difference weight respectively, and exp(·) is the natural exponential function with base e;

[0139] S63. Sort all objects according to the priority function and load the urban object structure and the corresponding bound data in descending order to construct the three-dimensional view content required for the current frame;

[0140] S64. Locate the rendering sub-blocks according to the spatial encoding of the loaded objects and organize the spatial arrangement order of all rendering sub-blocks on the two-dimensional geographic projection plane to merge and form the complete urban-level three-dimensional view of the current frame.

[0141] By dynamically sorting the node loading priorities in combination with the scheduling sequence before each frame rendering and using time-driven, adjacent structure differences, and semantic differences as adjustment factors, the system can ensure the priority loading of key views while compressing the loading frequency of non-critical data and improving the frame-to-frame response speed. The finally constructed urban-level three-dimensional view has a clear structure and reasonable organization, and can achieve a continuous rendering experience with low latency and high frame rate.<f

[0142] Example 1:

[0143] To verify the feasibility of the present invention in implementation, the present invention is applied to the intelligent urban management platform of a coastal provincial capital city. The planned area of this city reaches 1200 square kilometers, and the total data volume of the 3D CIM model exceeds 3.2TB, including approximately 420,000 building objects, 20,000 geospatial entities, 11 types of Internet of Things perception data and remote sensing image data, and real-time dynamic data such as traffic flow, environmental monitoring, building energy consumption, and population distribution are synchronously accessed. Due to the wide scene range, diverse data types, and high update frequency, the original urban modeling platform has problems such as slow data loading, chaotic semantic information, and frame switching lag, seriously affecting the overall control of the urban operation status by management personnel and the decision-making response efficiency.

[0144] In the actual application process, first, the building information data, geographic information data, remote sensing images, and Internet of Things sensing data of the above-mentioned city are uniformly collected and standardized preprocessed, and spatial projection conversion, timestamp synchronization, and unified mapping of semantic tags are completed. On this basis, the method proposed by the present invention constructs a city object structure for all 420,000 building objects, integrating their 3D geometric boundaries, administrative division positioning, semantic function attributes, and perception time nodes, providing a standardized data basis for subsequent graph structure construction and scheduling strategies.

[0145] Subsequently, the quadtree space partitioning algorithm is used to logically partition the 3D space of the entire city. Finally, the city is divided into approximately 65,536 rendering sub-blocks of fixed size, and the size of each sub-block is controlled within the range of 64m×64m, and the rendering index depth is set to 8 layers. Each rendering sub-block establishes a mapping with the bound city object structure through a unique spatial code, forming a multi-modal fusion network integrating semantic tag graph structure and spatial index. By calculating the semantic similarity and spatial adjacency relationship between objects, more than 310,000 semantic edges are constructed to express the potential connections between buildings in terms of function, location, and time dimensions, significantly improving the organization efficiency of the model and the flexibility of rendering scheduling.

[0146] After the system is officially put into operation, the rendering units are dynamically scheduled through the frame-by-frame rendering cost function and the frame-level optimal path search algorithm. The platform conducts a comparative experiment on the scheduling response capabilities under different time periods and different interaction perspectives. The data shows that after accessing dynamic Internet of Things data, the rendering method proposed by the present invention can reduce the first-frame loading time of the urban 3D scene from the original 9.3 seconds to 2.7 seconds without sacrificing the visualization accuracy, and the average frame-by-frame rendering switching delay is shortened from the original 1.25 seconds to 0.43 seconds, and the overall resource occupancy of the system is reduced by about 37%. In the response test conducted during the peak period of user interaction (such as 9:00-10:00 am and 15:00-17:00 pm every day), the present invention can achieve an average rendering efficiency of 96 frames per minute, while the original system can only maintain at 41 frames per minute.

[0147] In actual business scenarios, such as in the emergency linkage simulation of urban drainage systems, the platform drives the rendering module in real time based on the perception data, automatically and preferentially schedules the loading of regional sub-blocks containing high-risk waterlogging points, and combines the information of key areas such as "sunken squares" and "tunnel entrances" in the semantic map for high-precision rendering and pushing. This ensures that the emergency command department can complete the deployment of the three-dimensional situation map within 60 seconds, shortening the response time by approximately 72% compared to the original system. The following is a performance comparison table sorted based on actual test data:

[0148] Table 1 Performance Verification Data Table of the Urban-level CIM Streaming Rendering Method Based on Multimodal Fusion

[0149]

[0150] As can be seen from the above embodiments, the application of the present invention in the actual CIM platform not only solves the core problems such as difficult multi-source data fusion, complex spatial structure organization, and low rendering efficiency in terms of technical implementation, but also significantly improves the visualization quality and response speed of the system in terms of application effects, and has good promotion value and implementation ability.

[0151] The above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered by the protection scope of the present invention.

Claims

1. A method for streaming rendering of urban-level CIM scenarios based on multimodal fusion, characterized in that, It includes the following steps: S1. Collect urban data and perform preprocessing; S2. Based on the preprocessed urban data, establish cross-modal entity alignment relationships, and construct an urban object structure including geometric features, semantic labels, and timestamp information; S3. Use the quadtree method to perform spatial partitioning on the geometric features in the urban object structure, map the partitioned spatial regions to rendering sub-blocks of a fixed size, and assign unique spatial encodings; S4. Bind the rendering sub-blocks to the urban object structure, construct a multi-modal fusion network with a semantic label graph structure, and generate a set of multi-modal rendering units with spatial and semantic encodings; S5. Combine timestamp information to construct an inter-frame rendering cost function, and use a frame-level optimal path search algorithm to schedule and sort the set of multi-modal rendering units to form an inter-frame scheduling sequence; S6. Before the start of each frame rendering, determine the rendering sub-blocks corresponding to the current frame in combination with the inter-frame scheduling sequence, and load the rendering units with the highest priority to form a city-level 3D view of consecutive frames; S7. Based on the city-level 3D view, update the inter-frame scheduling sequence, dynamically adjust the loading order of the rendering sub-blocks and rendering units of subsequent frames, and achieve stable and efficient streaming rendering of the city-level CIM scene.

2. The method for streaming rendering of urban-level CIM scenarios based on multimodal fusion according to claim 1, wherein The urban data includes building information data, remote sensing image data, and Internet of Things perception data, and the preprocessing includes spatial projection conversion, timestamp alignment, and semantic label mapping.

3. A method for streaming rendering of urban-level CIM scenarios based on multimodal fusion according to claim 1, characterized in that The semantic labels are extracted by parsing the classification fields in the remote sensing image data, building information data, and Internet of Things perception data, and a unique mapping relationship is established with the corresponding geometric features and timestamp information to form a semantic description vector that can be used for rendering scheduling.

4. A method for streaming rendering of urban-level CIM scenarios based on multimodal fusion according to claim 1, characterized in that, The semantic label graph structure is a weighted undirected graph. Each node in the graph corresponds to an urban object structure. The edge weight of each edge is jointly calculated by the cosine similarity between semantic label vectors and the spatial coexistence relationship. The edge weight is mapped using a logical normalization function to enable the semantic label graph to have differentiable, clusterable, and propagable embedding representation capabilities, which is used to enhance the connectivity and expression integrity of multi-modal rendering units at the semantic level.

5. A method for streaming rendering of urban - level CIM scenarios based on multimodal fusion according to claim 1, wherein, The specific content of S2 includes: S21. Extract three-dimensional geometric structure parameters based on the preprocessed building information data. The structure parameters include building height, boundary contour, and outer facade grid point set, denoted as set G = {g1, g2,..., g m}, where g i represents the geometric feature vector of the i-th building, and m is the number of buildings; S22. Perform a reference check on the pre-processed remote sensing image data and the building information data, extract semantic description tags, where the tags include building type, functional attributes, and construction status, and construct a semantic tag set S = {s1, s2,..., s m}, where s i represents the semantic tag vector corresponding to the i-th building; S23. Based on the timestamp information in the Internet of Things perception data, construct a multi-source time synchronization sequence T = {t1, t2,..., t m}, where t i represents the time node associated with the i-th building; S24. Based on the geometric feature vector g i , semantic label vector s i and time node t i , aiming to minimize the inter-modal difference degree, establish a cross-modal entity alignment mapping relationship; S25. Combine g i , s i with t i to form the urban object structure O i = (g i , s i , t i ), and construct the set of urban object structures O = {O1, O2,..., O m} for subsequent rendering unit generation and scheduling.

6. A method for streaming rendering of urban-level CIM scenarios based on multimodal fusion according to claim 1, characterized in that The specific content of S3 includes: S31. Extract the geometric feature vector g in each structure from the set of urban object structures i , where the g i includes the six - tuple of the boundary coordinates of the building where represents the minimum coordinate value of the i - th object in the X - axis direction, represents the maximum coordinate value of the i - th object in the X - axis direction, represents the minimum coordinate value of the i - th object in the Y - axis direction, represents the maximum coordinate value of the i - th object in the Y - axis direction, represents the minimum coordinate value of the i - th object in the Z - axis direction, represents the maximum coordinate value of the i - th object in the Z - axis direction, and m is the number of buildings; S32. Construct a two-dimensional geographic projection plane, use the horizontal projection center of the building as a reference point, establish a global rendering bounding box B, and form a root node area; B = [X min , X max , Y min , Y max ; Traverse i = 1, 2,..., m, where max represents the maximum value function and min represents the minimum value function; S33. Set the minimum rendering sub-block size to d0, and terminate the division process when the width or height of any sub-region is less than d0, generating a set of rendering sub-blocks Q = {q1, q2,..., q k}, where q j represents the j-th leaf node rendering sub-block, and k is the number of rendering sub-blocks; S34. Assign a unique spatial encoding C to each rendering sub-block q j where the spatial encoding C j is constructed according to the depth-first order and the node path generation rule: j ​ Among them, C j represents the spatial encoding of the j-th rendering sub-block, D represents the depth of the rendering sub-block in the quadtree, and I n ∈ {0, 1, 2, 3} represents the quadrant index of the n-th layer in the path from the root node to the j-th leaf node. The quadrant index is numbered from 0 for the first quadrant to 3 for the third quadrant in the clockwise direction; S35. Project the geometric feature g in each urban object structure O i onto the geographic plane, and determine whether the boundary coordinates fall within the spatial region covered by the rendering sub-block q i . If the spatial inclusion relationship is satisfied, bind the urban object structure O j to the corresponding rendering sub-block q i and record the spatial code. j ​ 7. A method for streaming rendering of urban-level CIM scenarios based on multimodal fusion according to claim 6, characterized in that The root node area is constructed through the bounding box B and is recursively partitioned using a quadtree partitioning strategy; Among them, represents the spatial range of the j-th area on the (n + 1)-th layer, are the boundary coordinate values of the j-th area, that is, the coordinates of the east-west boundary and the north-south boundary.

8. A method for streaming rendering of urban-level CIM scenarios based on multimodal fusion according to claim 1, characterized in that The specific content of S4 includes: S41. Based on the set of rendering sub - blocks Q = {q1, q2,..., q k}, and its spatial encoding set C = {C1, C2,..., C k}, distribute each structure O m in the set of urban object structures O = {O1, O2,..., O i} = (g i , s i , t i ) to the corresponding rendering sub - block q j , and record the binding relationship pair (O i , C j ), where k is the number of rendering sub - blocks and m is the number of buildings; S42. Based on the semantic tag set S = {s1, s2,..., s m}, construct the semantic tag adjacency matrix A s , and use the similarity function to calculate the semantic adjacency weights between any two city objects: Among them, A s (i,j) represents the urban object structure O i and O j in the semantic dimension of the adjacency weight, cos(s i , s j ) represents the cosine similarity of the semantic label vectors s i and s j , δ ij = 1 indicates that the urban object structure O i and O j are in the same rendering sub-block, δ ij = 0 indicates that they are in different rendering sub-blocks, α and β are semantic similarity weight parameters, and exp(·) is the natural exponential function with base e; S43. Construct the multi-modal fusion graph structure G f =(V, E), where the node set V = {v1, v2,..., v m}, and each node v i represents the urban object structure O i . The edge set E contains two types of edges: geometric adjacency edges and semantic label edges. The formula for calculating the fusion edge weight is as follows: Among them, W(i,j) represents the edge weight between nodes v i and v j in the fusion graph, ‖g i - g j ‖2 represents the Euclidean distance between geometric feature vectors, and ‖g i ‖2 represents the second norm of the geometric feature vector g i . A s (i,j) is the corresponding element in the semantic adjacency matrix, λ1 and λ2 are two types of edge weight distribution coefficients, and max represents the maximum value function; S44. Generate a set of multimodal rendering units U = {u1, u2,..., u k} based on the spatial encoding of the multimodal fusion graph structure and binding, where each rendering unit u j contains the spatial encoding C j of the rendering sub-block, the set of indices of the city object structure to which it belongs and the corresponding fusion graph sub-graph structure for inter-frame scheduling and graphics rendering.

9. A method for streaming rendering of urban-level CIM scenarios based on multimodal fusion according to claim 1, characterized in that, The specific content of S5 includes: S51. Extract the set I of the structure indices of the city objects corresponding to each rendering unit from the set of multi-modal rendering units j , and obtain the corresponding multi-source time synchronization sequence T = {t1, t2,..., t m}, where each rendering unit u j includes the spatial encoding C of the rendering sub-block j , t i represents the time node associated with the i-th building, k is the number of rendering sub-blocks, and m is the number of buildings; S52. Construct an inter-frame rendering cost function for measuring the switching overhead between any two rendering units u i and u j in consecutive frames: Among them, F(i, j) represents the inter-frame cost from rendering unit u i to u j ; |I i ∩I j | represents the number of shared objects in the two units; t p is the timestamp of the urban object structure O p ; τ is the reference time of the current frame; s p is the semantic label vector; is the average value of the semantic label vectors contained in the rendering unit u j ; η1, η2, and η3 are the weighting coefficients of the structural difference, time difference, and semantic difference; exp(·) is the natural exponential function with base e; S53. Construct a frame-level optimal path search model, and set the goal to minimize the total rendering cost of the loading order of all rendering units within consecutive frames; where min represents the minimum value function, π = {π1, π2,..., π k} represents the access order of rendering units from the first frame to the k-th frame, k is the number of rendering sub-blocks, Π represents the set of all access paths, F(π n , π n+1 ) represents the rendering cost between two consecutive frames, represents the set of object indices included in the n-th frame rendering unit, μ is the path smoothness regulation factor, and the path similarity term is used to penalize excessive changes in the object set between consecutive frames; S54. Obtain the optimal scheduling path π according to the search results * , and use the rendering unit sequence corresponding to each frame in the path as the inter-frame scheduling sequence to control the loading order and time distribution of the multi-modal rendering units.

10. A method for streaming rendering of urban-level CIM scenarios based on multimodal fusion according to claim 1, characterized in that, The specific content of S6 includes: S61. Based on the inter-frame scheduling sequence π * ={π1, π2,..., π k}, determine the multi-modal rendering unit to be loaded when the current frame index is n and obtain the rendering sub-block encoding bound to the rendering unit and the set of urban object structure indices S62. Extracting rendering units based on the multimodal fusion graph structure G f , and extracting the set of sub-graph nodes contained in and sorting the nodes according to the priority function: Among them, V represents the set of nodes, and P(i) represents the urban object structure O i The rendering priority value in the current frame, t i Represents the time node associated with the i-th building, τ n Is the reference time of the current frame, Is the urban object structure O i In the graph structure G f The set of adjacent nodes in, Is the set of object index loaded in the previous frame, s i Is the urban object structure O i The semantic label vector of, Is the mean value of all semantic vectors of the current rendering unit. γ, θ1, θ2, and θ3 are the time change rate adjustment coefficient, time-driven weight, adjacent difference weight, and semantic difference weight respectively. exp(·) is the natural exponential function with e as the base; S63. Sort all objects in according to the priority function, load the urban object structure and the corresponding binding data in descending order, and construct the 3D view content required for the current frame; S64. Locate the rendering sub-blocks according to the spatial encodings of the loaded objects, and organize the spatial arrangement order of all rendering sub-blocks on the two-dimensional geographic projection plane, and merge them to form a complete city-level 3D view of the current frame.

Images