Port avionics hub digital twinborn multi-level simulation model construction and visualization method and system

Abstract

The invention discloses a port avionics hub digital twin multi-level simulation model construction and visualization method, which comprises the following steps: obtaining a multi-view RGB image of a target port area, obtaining a camera attitude of each view by using an SfM algorithm, and reconstructing a sparse three-dimensional point cloud; constructing an LOD2 shell model with a semantic mark; an LLO3 model with holes is constructed; the LOD0-LOD3 model is serialized into a glTF 2.0 binary format and transmitted to a client side through HTTP / 2, the client side calls WebGL 2.0 through a browser to complete decompression, uploading and PBR rendering, and interactive plug-in-free visualization is achieved; constructing an eight-fork-two-fork-four-fork mixed spatial index, and performing multi-stage division and persistence on the LOD < 0 >-LOD < 3 > tiles, so that a client locates visible tiles within constant time based on node numbers; visual cone cutting and screen space error judgment are adopted to execute progressive loading on visible tiles, LOD0 is loaded at first, then LOD1-LOD3 is subjected to incremental superposition, low-level tiles in the same area are replaced with the LOD3 in a cross fading mode at the ultra-close distance (the Euclidean distance d from the center of a camera to the center of a target tile is smaller than or equal to 100 m), and smooth visual experience is obtained in a browser.

Description

Technical Field

[0001] This invention relates to the field of modeling and visualization, specifically to a method and system for constructing and visualizing a multi-level digital twin simulation model of a port and shipping hub. Background Technology

[0002] With the rapid development of the global economy, port and shipping hubs, as important transportation and energy conversion nodes, are facing increasing complexity in their design, construction, and operation. Digital twin technology, as an emerging modeling and analysis tool, provides powerful data support and analytical capabilities for the entire lifecycle management of physical structures by creating virtual models of them. Model building is a crucial part of digital twins. A port system is a complex man-made system encompassing multiple stages, operating according to a set of man-made rules. The complexity of these management rules and the diversity of the numerous system elements present significant challenges to port modeling.

[0003] Traditional mathematical modeling methods are increasingly proving inadequate for solving port problems. Driven by the development of artificial intelligence (AI) technology, a number of new modeling techniques and methods for complex large-scale systems have been developed and successfully applied in numerous engineering practices. Modeling techniques for complex systems, including hybrid modeling, HLA-based modeling, hierarchical-aggregational modeling, and modular-integrated modeling, have been widely applied in transportation fields such as ports. For example, Jin and Wang developed a port multimodal transport simulation model based on an HLA distributed simulation environment. The simulation federation consists of an operational federation and a coordination and management federation. The operational model uses a discrete event system simulation model to simulate local logistics operation subsystems such as ports, highways, and railways, while the coordination and management federation coordinates operations based on the state of the operational federation. Fotuhi et al. established an agent-based simulation model to simulate the operation of terminal yard cranes. Each crane acts as an autonomous agent seeking to maximize its utility. The model can capture the basic decision-making attributes of crane operators when selecting the next truck to serve. Zheng Liyu applied the hierarchical modeling method of DEVS to the modeling process of a petrochemical logistics system simulation model. A logistics simulation platform for an oil refinery was developed, and its effectiveness was verified. Gao Wei developed a hierarchical and dynamic model of a container terminal logistics system using Petri net modeling, and then implemented the simulation model on a simulation platform. Peng Gao et al. established a distributed federation simulation model for a container terminal multimodal transport system. The simulation federation includes an operations federation, an information management federation, and an operations management federation. Zhou Wenting developed a distributed container simulation platform based on HLA, and established a simulation model consisting of federation members such as a management information system, a quay crane operation system, a yard operation system, and a gate operation system. Although the above research has made significant progress in the construction of digital twin models, existing research is still relatively insufficient in the automated modeling of large-scale models.

[0004] The importance of automated modeling is self-evident when dealing with large-scale models. It not only significantly improves modeling efficiency and reduces the tedium of manual operations, but also ensures the accuracy and consistency of the model. However, existing modeling methods often rely heavily on manual operation and expert knowledge, which proves inadequate when dealing with large and complex systems. The challenges of automated modeling are particularly pronounced in multimodal transport systems involving multiple subsystems, various modes of transport, and complex interactions.

[0005] With the booming development of the computer industry, and leveraging advanced internet technologies, especially WebGL, digital twin models of large ports can be efficiently displayed through web browsers. This has become an important trend in the development of digital twin technology for large-scale engineering projects. Deeply integrating digital twins with WebGL technology to build a Web-based digital twin engineering visualization cloud platform not only helps users break free from the constraints of professional software platforms and clients, lowering the application threshold of digital twin technology, but also enables real-time cross-platform data sharing via the internet, thereby promoting the application of digital twin technology in a wider range of fields. However, as the complexity and detail of digital twin models continue to increase, the efficiency of model transmission and rendering on the network becomes increasingly prominent. To reduce computational overhead, commonly used lightweight rendering methods include culling optimization, LOD algorithms, MergeMesh, and InstancedMesh. For example, Su et al. proposed a two-layer visibility culling algorithm based on scene structure and view frustum culling, fully utilizing the high precision of binary tree spatial subdivision to improve rendering performance. Gan et al. improved the hierarchical tree structure from top to bottom into AABB and OBB, using synchronous descent rules to achieve node traversal, and adopted the triangle area weighting method to improve the calculation method of OBB center points, thus improving the accuracy of range queries. Lee et al. proposed a scalable online culling algorithm, using an object-level bounding volume hierarchy to improve the traditional rasterization occlusion culling method, which outperforms existing solutions in terms of scalability, culling efficiency, and occlusion query performance. LOD algorithms can be divided into discrete LOD algorithms and continuous LOD algorithms based on simplified continuous types. Discrete LOD algorithms pre-load models of multiple detail levels in stages, making it impossible to smoothly transition between two detail levels. Continuous LOD algorithms can generate 3D models with continuous detail, allowing for smooth transitions when switching between different detail levels. Schütz et al. proposed a real-time continuous LOD method for point clouds. Based on camera coordinates, orientation, and distance to points, vertex buffers are repeatedly created from the complete point cloud to update the rendering results point-by-point, achieving a smooth transition between point clouds of different densities. Li et al. proposed a topological data structure based on continuous LOD, setting the correspondence between the geometric data of 3D building models at different LOD detail levels, constraining the edges and faces of triangular meshes in the 3D building model, and defining the state of LOD geometric data according to different stages of transformation to achieve continuous LOD transformation of the building model. However, in terms of rendering efficiency, dynamic data scheduling, and multi-level detail management of large-scale models, there are still problems such as low rendering efficiency and low display quality. There is an urgent need to improve the rendering efficiency of large-scale models to ensure efficient data transmission, smooth rendering, and real-time sharing. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method and system for constructing and visualizing a multi-level digital twin simulation model of a port and shipping hub. Addressing the low level of automation in existing port modeling research, this invention provides an end-to-end automated modeling framework. It uses simplified methods to encapsulate architectural planar primitives to accurately represent the geometry of buildings and generate high-quality LOD2 and LOD3 models. To address the issue of low transmission and rendering efficiency caused by the increasing complexity and detail of existing digital twin models, a multi-dimensional adaptive spatial index structure design is proposed to improve data query and visualization efficiency. Finally, a progressive loading data dynamic scheduling method is used to further optimize the model rendering process, further improving rendering efficiency and stability.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A method for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub includes:

[0009] S1. Acquire multi-view RGB images of the target port area, use the SfM algorithm for feature matching and epipolar constraint solving, obtain the camera pose of each view and reconstruct sparse 3D point cloud;

[0010] S2. Based on the sparse 3D point cloud, a planar primitive point cloud is generated by Euclidean clustering and RANSAC plane fitting. The planar primitive point cloud is orthogonally projected and then constructed as a semantically labeled LOD2 shell model by convex polygon extraction, vertex simplification and triangulation.

[0011] S3. Based on the camera projection matrix and epipolar geometry, the two-dimensional opening obtained by deep learning semantic segmentation is mapped to three dimensions and Boolean difference removal is performed to automatically upgrade to obtain a LOD3 model with holes;

[0012] S4. Serialize the LOD0-LOD3 models into glTF 2.0 binary format and transmit them to the client via HTTP / 2. The client uses a browser to call WebGL 2.0 to complete decompression, uploading, and PBR rendering, achieving interactive, plugin-free visualization. The LOD0-LOD3 models are the multi-level simulation models of the port's navigation and power hub digital twin.

[0013] S5. Construct a hybrid spatial index of octagonal, binary, and quadrangular branches, perform multi-level partitioning and persistence of LOD0-LOD3 tiles, enabling clients to locate visible tiles in constant time based on node numbers, reducing data input / output and transmission overhead;

[0014] S6. The visible tiles are progressively loaded using view frustum clipping and screen space error determination. First, LOD0 is loaded, then LOD1-LOD3 are incrementally superimposed. At very close distances, LOD3 is used to replace the lower-level tiles in the same area with LOD3 in a cross-fading manner, so as to obtain a smooth visual experience in the browser. The very close distance refers to the Euclidean distance d ≤ 100m from the center of the camera to the center of the target tile.

[0015] Furthermore, in step S1, a drone is used to fly over the target port area and automatically capture multi-view RGB images covering the entire port area. The SfM algorithm is used to extract and match features from the multi-view RGB images. Combined with epipolar geometry constraints, the camera pose of each RGB image is automatically calculated. The camera pose includes rotation matrix and translation vector. The matched feature points are triangulated to reconstruct a sparse 3D point cloud of the building scene.

[0016] Furthermore, in step S2, the sparse 3D point cloud is first grouped using the Euclidean clustering algorithm, and the planar primitive point cloud is extracted for each group using RANSAC plane fitting. The planar primitive point cloud is orthogonally projected onto the fitting plane, and then subjected to Graham convex polygon extraction, Douglas-Peucker vertex simplification, and ear-cutting triangulation to generate triangular meshes with semantic labels as roof, facade, and slope. After merging, a LOD2 shell model is formed. The semantic labels are classified based on the angle θ between the plane normal and the Z-axis of the world coordinate system. θ≤30° is labeled as roof, θ≥60° is labeled as facade, and the rest are labeled as slope. The simplification threshold ε for convex polygons during vertex simplification is set to 0.05 m~0.15 m.

[0017] Furthermore, in step S3, based on the obtained LOD2 shell model, the camera projection matrix output by the SfM algorithm is used to perform deep learning semantic segmentation on the facade image to obtain the opening mask; the mask corner points are mapped to adjacent views and triangulated through epipolar geometry to reconstruct the three-dimensional opening corner points; Boolean difference truncation is performed on the LOD2 mesh to insert the opening geometry and output the LOD3 model with holes.

[0018] Furthermore, in step S4, depth testing, backface culling, and 4×MSAA are enabled on the browser side, and rotation, scaling, and roaming interactions are achieved through a trackball algorithm.

[0019] Furthermore, in step S5, the process of constructing the octagonal-binary-quadrilateral hybrid spatial index is as follows:

[0020] The root node is initialized with full-scene AABB; the first level uses an octree to divide the main buildings; the second level uses a binary tree to divide the layers along the elevation; the third level uses a quadtree to subdivide the equipment area in the plane; the leaf node stops when the side length or number of tiles reaches the threshold. Node information is written to the disk database in the structure of "node number-parent node number-level-bounding box-tile handle list" and stored for a long time. The browser can locate visible tiles in constant time by using the node number, reducing data input / output and transmission overhead.

[0021] Furthermore, in step S6, each rendering frame removes invisible branches by frustum culling; the screen space error e is calculated in real time for visible tiles and compared with a threshold ε: if e>ε, a higher-level tile is incrementally loaded; if e<ε, it is retained or downgraded; initially only LOD0 is loaded, then LOD1-LOD3 are superimposed. When the tiles are very close, LOD3 is used to replace the lower-level tiles in the same area with LOD3 in a cross-fading manner, achieving a smooth transition with addition as the main method and replacement as the auxiliary method, reducing the GPU rendering burden and maintaining a 60 fps visual experience.

[0022] This invention also provides a device for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub, comprising:

[0023] The image acquisition and attitude calculation module is used to acquire multi-view RGB images of the target port area, use the SfM algorithm for feature matching and epipolar constraint solving, obtain the camera attitude of each view and reconstruct sparse 3D point cloud.

[0024] The geometric modeling module is used to generate a planar primitive point cloud based on the sparse 3D point cloud through Euclidean clustering and RANSAC plane fitting, and to construct a semantically labeled LOD2 shell model by orthogonally projecting the planar primitive point cloud and sequentially extracting convex polygons, simplifying vertices and triangulating it.

[0025] The opening reconstruction and Boolean difference removal module is used to map the two-dimensional openings obtained from deep learning semantic segmentation to three dimensions based on the camera projection matrix and the outer pole geometry, and perform Boolean difference removal to automatically upgrade to a LOD3 model with holes.

[0026] The formatting and transmission module is used to serialize LOD0-LOD3 models into glTF 2.0 binary format and transmit them to the client via HTTP / 2;

[0027] The client-side rendering module is used to complete decompression, uploading, and PBR rendering in the WebGL 2.0 environment, enabling interactive, plugin-free visualization.

[0028] The hybrid spatial index module is used to build an octagonal-binary-quadrilateral hybrid index, which performs multi-level partitioning and persistence of LOD0-LOD3 tiles, enabling clients to locate visible tiles in a constant time based on node numbers, reducing data input / output and transmission overhead.

[0029] The progressive loading module is used to progressively load visible tiles using view frustum clipping and screen space error determination. It first loads LOD0, then incrementally overlays LOD1-LOD3, and replaces lower-level tiles in the same area with LOD3 using crossfading at very close distances, resulting in a smooth visual experience in the browser. "Very close distance" refers to the Euclidean distance d ≤ 100m from the center of the camera to the center of the target tile.

[0030] Preferably, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for constructing and visualizing the digital twin multi-level simulation model of the port navigation and power hub.

[0031] Preferably, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method for constructing and visualizing the digital twin multi-level simulation model of the port navigation and power hub.

[0032] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0033] 1. Improve the automation level of modeling, reduce manual intervention, and correct workload; automatically calculate camera pose and generate point clouds through the SfM algorithm, and directly generate building outlines and geometry by combining automatic planar extraction and linear optimization, significantly reducing manual surveying and modeling steps. The automated end-to-end process improves efficiency and consistency. It solves the problem that traditional methods rely heavily on manual modeling and expert experience, which is time-consuming and labor-intensive when dealing with large-scale port areas.

[0034] 2. Generate high-quality LOD2 / LOD3 models, restoring details such as building openings; facade / opening segmentation based on deep learning, utilizing registered images and camera epipolar geometry for opening point triangulation. Deep learning accurately identifies opening masks on the image plane, and triangulation combining two views and projection matrices accurately projects the 2D segmentation results back to 3D, thereby restoring details such as doors, windows, and openings at the LOD3 level, improving model realism and semantic usability. This addresses the problem that LOD2 models typically only have a general geometric shape and lack openings, making them difficult to use for detailed visualization and simulation.

[0035] 3. This invention alleviates network transmission and browser rendering pressure, enabling web visualization of large-scale models. It combines multi-dimensional adaptive spatial indexing, tile generation, progressive loading, and WebGL rendering. Spatial indexing quickly locates and requests only the necessary tiles; progressive loading uses screen space error and view frustum judgment to retrieve only tiles with appropriate LOD; and WebGL renders directly on the browser side. This combination significantly reduces transmission volume and GPU rendering burden, achieving smooth display. It solves problems such as stuttering, memory bottlenecks, or bandwidth limitations caused by network transmission and browser rendering of large-scale models.

[0036] 4. Improve data query and visualization response speed; the hierarchical index structure adopts root → octree → binary tree → quadtree → quadtree leaf. The mixed use of different tree structures makes the index perform well in both coarse-to-fine and planar to elevation directions, which can quickly filter target tiles, floors or individual components, improve positioning speed and scheduling efficiency; and solve the problem that a single index structure cannot simultaneously adapt to horizontal spatial, elevation distribution and component-level fine-grained queries.

[0037] 5. Reduce GPU instantaneous load and improve rendering stability; dynamic scheduling selects tiles to draw based on GPU performance, tile priority, visibility, and screen space error; a "primarily adding, secondarily replacing" LOD transition strategy. Dynamic scheduling monitors the load at runtime and limits the number of high LOD tiles loaded simultaneously; progressive loading and priority ensure that high precision is loaded only when necessary, reducing instantaneous memory and drawing pressure, and smoothing frame rate fluctuations; resolving issues such as sudden FPS drops or memory overflows caused by instantaneously loading a large number of high-precision meshes.

[0038] 6. The cross-platform, client-free visualization experience on the web lowers the barrier to entry; WebGL natively supports mainstream browsers, and combined with tiled rendering and progressive loading, it enables interactive 3D viewing in ordinary terminal browsers, reducing the barrier to entry and facilitating data sharing. It solves problems such as the complexity of deploying professional desktop software and user limitations imposed by the client environment.

[0039] 7. Improve the semantic usability of models to support subsequent simulation, scheduling, and operation and maintenance applications; LOD3 includes opening information, and LOD4 contains high-precision single-unit models that can be bound with semantic information; binding semantics (such as device ID, attributes, and functions) to high-precision single units can directly drive simulation or business logic, improving the practical value of digital twins. This addresses the difficulty of directly participating in simulation, collision detection, and device scheduling applications with no or low semantics.

[0040] 8. Improve scalability and ease of subsequent maintenance; hierarchical tiles and indexes, incremental loading, and tile replacement mechanisms reduce maintenance costs and support gradual iterative model improvement. Solve the problem of difficulty in maintaining and updating a one-time full deployment. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the method and process for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub in a specific embodiment.

[0042] Figure 2 This is a schematic diagram of the progressive loading principle. Detailed Implementation

[0043] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0044] Example 1

[0045] This embodiment provides a method for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub, as detailed below:

[0046] S1. Use a drone to fly over the target port area and automatically capture a set of RGB images from multiple views. Ensure that the captured images cover the entire port area and have sufficient resolution to extract detailed geometric information. Import the captured RGB image set into the SfM algorithm. Use the SfM algorithm to extract and match features from the multi-view RGB images. Combined with epipolar geometric constraints, automatically calculate the camera pose (rotation matrix and translation vector) for each image. Then, based on these camera poses, triangulate the matched feature points to reconstruct a sparse 3D point cloud of the building scene.

[0047] S2. Based on the sparse 3D point cloud data obtained from the SfM algorithm, the Euclidean clustering spatial clustering algorithm is used to initially group the sparse 3D point cloud. The RANSAC algorithm is used to perform plane fitting on each group of point clouds to extract the main planar primitives of the building (such as walls, roofs, and ground). For each group of planar primitive point clouds obtained by RANSAC fitting, the point cloud is first orthogonally projected onto its fitting plane, and the Graham scan algorithm is used to obtain planar convex polygons. Then, the Douglas-Peucker algorithm is used to compress the convex polygons with a threshold ε∈[0.05m,0.15m] to form simplified polygons. Next, the ear-cutting method is used to perform non-self-intersecting triangulation on the simplified polygons to generate triangular meshes that only represent the building's exterior walls or roofs. Finally, the angle θ between the plane normal vector and the world coordinate system Z-axis is calculated. If θ≤30°, it is marked as "roof", θ>60° is marked as "facade", and the rest are marked as "sloping face". The semantic tags are written into the attribute fields of the corresponding triangular facets to complete the automatic construction of the LOD2 geometric model.

[0048] S3. To upgrade the acquired LOD2 model to a LOD3 model containing building opening information, firstly, in the image sequence registered by the SfM algorithm, the opening area is segmented using the corresponding camera projection matrices P and P′. Then, based on the principle of epipolar geometry, the segmented two-dimensional opening corner points x and x′ are triangulated into three-dimensional space to reconstruct the three-dimensional coordinates of the opening corner point X. Specifically, the coordinates of the three-dimensional opening corner point X are solved using the following formula:

[0049] (1)

[0050] Where λ and λ' are constants representing the position parameters of point X on the corresponding camera ray, this system of equations is solved by singular value decomposition (SVD) to obtain the three-dimensional coordinates of X in the world coordinate system. To obtain the two-dimensional opening corner points x and x', this invention uses a trained deep learning model (TernausNet) to perform semantic segmentation of the facade in the first view image (i.e., the calibrated input image used for opening semantic segmentation, whose camera projection matrix is ​​P). The training set consists of 12,000 4K RGB facade images (lateral overlap ≥60%, flight altitude 80 m, and desensitized) collected by 5 port UAV flights. The annotation specifications define three types of pixel-level labels: "opening corner points / opening interior / facade background". The labelme polygon tool is used for two-person cross-annotation and sampling inspection until the pass rate is ≥95%. Finally, 10,500 images and corresponding PNG masks are retained. The input resolution is 512×512. The pixel data augmentation includes horizontal flipping, brightness ±10%, and Gaussian blur σ=1; the key hyperparameters are Adam optimizer initial learning rate 1e-4, batch size 16, Dice+Focal weighting 1:1, training for 120 epochs until the validation set Dice≥0.85; the softmax probability threshold during inference is 0.5. Each pixel of the input RGB image is classified into three semantic categories: “opening corner point”, “opening interior”, and “facade background”, generating three types of binary masks: opening corner, opening region and facade. Then, using the homography constraint of the opening in the planar structure, the opening corner point x detected in the first view is mapped to the second view to obtain the corresponding opening corner point x′. Substituting x and x′ into equation (1), the corresponding three-dimensional opening corner point X can be calculated, realizing the automatic projection and precise positioning of opening information from two-dimensional image to three-dimensional model, thereby completing the construction of LOD3 model.

[0051] S4. On the server side, serialize the LOD0-LOD3 models in glTF 2.0 binary format (.glb), and transmit the .glb file in chunks to the client via HTTP / 2 protocol. The browser uses a JavaScript decoding library to decompress the mesh data in parallel. The WebGL 2.0 API is called to perform the following steps in sequence: create a GLBuffer and upload vertex and index data to GPU memory; compile and link the vertex shader and fragment shader, with the vertex shader implementing model-view-projection matrix transformation and the fragment shader implementing PBR lighting calculation; set depth testing, backface culling, and multisampling anti-aliasing (MSAA 4×) states; in the rendering loop, use requestAnimationFrame() to implement 60... The FPS refresh rate converts user mouse / touch events into quaternions using the Arcball algorithm, updating the model rotation matrix in real time. Scroll wheel events are normalized to adjust the camera's field of view, achieving smooth scaling. Ultimately, users can perform 360° rotation, stepless scaling, and panning operations on the large port digital twin model within a web browser without installing any plugins, obtaining a 3D visual experience consistent with local rendering.

[0052] S5. See Figure 2 Based on an improved spatial indexing algorithm, data from a large-scale port digital twin model is indexed and organized. By optimizing the spatial index structure, data query and visualization efficiency is improved. According to the WebGL display requirements in step three, the spatial indexing algorithm supports data scheduling. By quickly locating the required data blocks, the overhead of data loading and transmission is reduced, improving visualization efficiency. Specifically, this includes:

[0053] S501. Initialize the root node. First, traverse the existing set of LOD0-LOD3 tiles, read the axis-aligned bounding box of each tile, calculate the maximum and minimum coordinates of all vertices in the X, Y, and Z directions, obtain the minimum hexahedron covering the entire scene, use it as the spatial range of the root node, and record the file handles pointing to all LOD0 tiles in the root node;

[0054] S502. First-level octree partitioning. Divide the root node hexahedron into two equal parts in each of the X, Y, and Z directions to form eight sub-cubes; for each sub-cube, perform intersection tests with the bounding boxes of all LOD1 tiles in turn—if two boxes overlap in all three axes, add the LOD1 tile handle to the candidate list of the sub-cube; after removing empty lists, generate first-level child nodes for the remaining sub-cubes and assign them unique node numbers, completing the first-level expansion of the octree.

[0055] S503. Second-level binary tree layering. For high-frequency access buildings such as multi-story warehouses and towers, check the height difference of LOD2 tiles within the first-level node: if the height difference exceeds the set layer height threshold, slice the tiles at equal intervals along the Z-axis with a fixed layer height to form several horizontal layers; regroup the LOD2 tiles according to the slices where their center elevation falls, generate a binary child node for each group, and record the bottom and top elevations of the layer to complete the second-level expansion of the binary tree;

[0056] S504. Three-level quadtree partitioning. Each level of horizontal slice is further divided into four equal parts on the plane to obtain four sub-pillars; LOD3 device tiles within the slice are assigned to the corresponding sub-pillars according to the center point position on the plane, non-empty sub-pillars generate quadtree child nodes, and the pillar boundaries are recorded to complete the third level expansion of the quadtree;

[0057] S505. Leaf Termination and Index Persistence. When the side length of a sub-pillar drops below a set threshold, or when only one tile remains inside, partitioning stops, and the sub-pillar is marked as a leaf node. All nodes are written to a standard relational database table in the order of "node number - parent node number - hierarchy - bounding box - tile handle list". The entire table is loaded into memory when the server starts. The WebGL client can locate the set of visible tiles in constant time by using the node number, achieving millisecond-level data scheduling.

[0058] S6. Employing progressive loading helps the GPU dynamically select tiles of appropriate visualization levels for rendering, further reducing the rendering burden. Progressive loading is essentially a frustum-based rendering optimization strategy that determines the resolution level to be loaded based on the distance between the 3D tiles within the frustum and the camera's viewpoint.

[0059] S601. Visibility Preprocessing: Before rendering each frame, the system first performs a top-down intersection test on the hybrid space index tree built in step four, using the six planes of the camera's view frustum as clipping bodies: only nodes that intersect with the view frustum are retained to form a list of potential visible tiles, and the rest of the branches are removed to reduce the amount of subsequent calculations.

[0060] S602. Screen Space Error Determination: For each tile in the potentially visible list, its screen space error 'e' is calculated in real time. When the camera viewpoint is closer, the number of models on the screen decreases, and the accuracy increases, requiring 3D tiles with a higher LOD level. When the camera viewpoint is farther away, the number of models on the screen increases, and the accuracy decreases, allowing the use of tiles with a lower LOD level to reduce rendering pressure. During system operation, the screen space error is used as the criterion for tile replacement, and its calculation principle is as follows: Figure 2 As shown. Figure 2In this context, r represents the tile geometric error (in meters), d is the distance from the viewpoint to the center point of the tile (in meters), fov is the viewing angle of the frustum (in rad), h is the screen height (in pixels), and e is the screen spatial error (in pixels). The relationship between e and r can be calculated using equations (2) and (3):

[0061] (2)

[0062] (3)

[0063] During the camera viewpoint change, the computer continuously calculates the screen space error e of the tile and compares it with a set threshold ε. When e > ε, the tile of the next LOD level is loaded to replace the current tile; when e < ε, the current tile is retained for rendering.

[0064] S603. Loading strategy that prioritizes addition and supplements replacement:

[0065] a) Initial frame: Only LOD0 geological tiles covering the entire scene are loaded, and the first packet data size is ≤5MB;

[0066] b) Mid-range: When e is greater than ε for the first time, incrementally insert LOD1 port shell tiles, while retaining the original LOD0 tiles, to achieve "overlay" refinement; c) Close range: Continue to insert LOD2 floor structure tiles, still adding to the original tiles to avoid scene flickering;

[0067] d) Ultra-close range: When the distance between the camera and the target is less than the set threshold D_th (100m) and e is continuously greater than ε, the low-level tiles in the same area are unloaded and replaced with LOD3 device tiles. The replacement process uses inter-frame crossfade (alpha from 0 to 1 takes 300ms) to ensure visual smoothness.

[0068] S604. GPU Upload and Rendering: All tiles to be loaded are compressed using Draco geometry and then streamed to the client via HTTP / 2. The decompression yields vertex and index data, which is written to a WebGL 2.0 Vertex Buffer Object (VBO). Depth testing, backface culling, and 4×MSAA are enabled to ensure smooth rendering at 60fps.

[0069] S605. Semantic Binding: After the LOD3 tiles are loaded, their semantic information (device type, number, operation and maintenance status) is written into the glTFextras field. The front-end JavaScript parses and injects it into the attribute panel in real time to achieve integrated model-data perception.

[0070] Example 2

[0071] Based on the same inventive concept, this application also provides a device for constructing and visualizing a multi-level digital twin simulation model of a port and shipping power hub, which can be used to implement the method described in the above embodiments, as described in the following embodiments. Since the principle of the device for constructing and visualizing a multi-level digital twin simulation model of a port and shipping power hub is similar to that of the method for constructing and visualizing a multi-level digital twin simulation model of a port and shipping power hub, the implementation of the device can refer to the implementation of the method for constructing and visualizing a multi-level digital twin simulation model of a port and shipping power hub, and will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0072] The embodiments of the present invention provide a specific implementation of a device for constructing and visualizing a multi-level digital twin simulation model of a port and shipping power hub, which is capable of realizing the construction and visualization of a multi-level digital twin simulation model of a port and shipping power hub. The specific implementation includes the following:

[0073] The image acquisition and attitude calculation module is used to acquire multi-view RGB images of the target port area, use the SfM algorithm for feature matching and epipolar constraint solving, obtain the camera attitude of each view and reconstruct sparse 3D point cloud.

[0074] The geometric modeling module is used to generate a planar primitive point cloud based on the sparse 3D point cloud through Euclidean clustering and RANSAC plane fitting, and to construct a semantically labeled LOD2 shell model by orthogonally projecting the planar primitive point cloud and sequentially extracting convex polygons, simplifying vertices and triangulating it.

[0075] The opening reconstruction and Boolean difference removal module is used to map the two-dimensional openings obtained from deep learning semantic segmentation to three dimensions based on the camera projection matrix and the outer pole geometry, and perform Boolean difference removal to automatically upgrade to a LOD3 model with holes.

[0076] The formatting and transmission module is used to serialize LOD0-LOD3 models into glTF 2.0 binary format and transmit them to the client via HTTP / 2;

[0077] The client-side rendering module is used to complete decompression, uploading, and PBR rendering in the WebGL 2.0 environment, enabling interactive, plugin-free visualization.

[0078] The hybrid spatial index module is used to build an octagonal-binary-quadrilateral hybrid index, which performs multi-level partitioning and persistence of LOD0-LOD3 tiles, enabling clients to locate visible tiles in a constant time based on node numbers, reducing data input / output and transmission overhead.

[0079] The progressive loading module is used to progressively load visible tiles using view frustum clipping and screen space error determination. It first loads LOD0, then incrementally overlays LOD1-LOD3, and at very close distances (Euclidean distance d≤100m from the camera center to the center of the target tile), it replaces the lower-level tiles in the same area with LOD3 using crossfading, resulting in a smooth visual experience in the browser.

[0080] Preferably, embodiments of this application also provide a specific implementation of an electronic device capable of implementing all steps in the method for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub as described in the above embodiments. The electronic device specifically includes the following:

[0081] Processor, memory, communications interface, and bus;

[0082] The processor, memory, and communication interface communicate with each other via a bus; the communication interface is used to realize information transmission between server-side devices, metering devices, and user-side devices.

[0083] The processor is used to call the computer program in the memory. When the processor executes the computer program, it implements all the steps in the method for constructing and visualizing the multi-level simulation model of the digital twin of the port navigation and power hub in the above embodiments.

[0084] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps in the method for constructing and visualizing a multi-level digital twin simulation model of a port and waterway hub as described in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps in the method for constructing and visualizing a multi-level digital twin simulation model of a port and waterway hub as described in the above embodiments.

[0085] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, hardware + program embodiments are relatively simple in description because they are fundamentally similar to method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0086] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0087] While this application provides method operation steps as shown in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the method can be executed in the order shown in the embodiments or drawings or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0088] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0090] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0091] This invention is not limited to the embodiments described above. The above description of specific embodiments is intended to illustrate and explain the technical solutions of this invention. The specific embodiments described above are merely illustrative and not restrictive. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of this invention, and these modifications all fall within the scope of protection of this invention.

Claims

1. A method for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub, characterized in that, include: S1. Acquire multi-view RGB images of the target port area, use the SfM algorithm for feature matching and epipolar constraint solving, obtain the camera pose of each view and reconstruct sparse 3D point cloud; S2. Based on the sparse 3D point cloud, a planar primitive point cloud is generated by Euclidean clustering and RANSAC plane fitting. The planar primitive point cloud is orthogonally projected and then constructed as a semantically labeled LOD2 shell model by convex polygon extraction, vertex simplification and triangulation. S3. Based on the camera projection matrix and epipolar geometry, the two-dimensional opening obtained by deep learning semantic segmentation is mapped to three dimensions and Boolean difference removal is performed to automatically upgrade to obtain a LOD3 model with holes; S4. Serialize the LOD0-LOD3 models into glTF 2.0 binary format and transmit them to the client via HTTP / 2. The client uses a browser to call WebGL 2.0 to complete decompression, uploading, and PBR rendering, achieving interactive, plugin-free visualization. S5. Construct a hybrid spatial index of octagonal, binary, and quadrangular branches, perform multi-level partitioning and persistence of LOD0-LOD3 tiles, enabling clients to locate visible tiles in constant time based on node numbers, reducing data input / output and transmission overhead; S6. The visible tiles are progressively loaded using view frustum clipping and screen space error determination. First, LOD0 is loaded, then LOD1-LOD3 are incrementally superimposed. At very close distances, LOD3 is used to replace the lower-level tiles in the same area with LOD3 in a cross-fading manner, so as to obtain a smooth visual experience in the browser. The very close distance refers to the Euclidean distance d ≤ 100m from the center of the camera to the center of the target tile.

2. The method for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub according to claim 1, characterized in that, In step S1, a drone is used to fly over the target port area and automatically capture multi-view RGB images covering the entire port area. The SfM algorithm is used to extract and match features from the multi-view RGB images. Combined with epipolar geometry constraints, the camera pose of each RGB image is automatically calculated. The camera pose includes rotation matrix and translation vector. The matched feature points are triangulated to reconstruct a sparse 3D point cloud of the building scene.

3. The method for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub according to claim 1, characterized in that, In step S2, the sparse 3D point cloud is first grouped using the Euclidean clustering algorithm. For each group, RANSAC plane fitting is used to extract planar primitive point clouds. The planar primitive point clouds are orthogonally projected onto the fitting plane and then subjected to Graham convex polygon extraction, Douglas-Peucker vertex simplification, and ear-cutting triangulation to generate triangular meshes with semantic labels as roof, facade, and slope. These meshes are then merged to form the LOD2 shell model. The semantic labels are classified based on the angle θ between the plane normal and the Z-axis of the world coordinate system. θ≤30° is labeled as roof, θ≥60° is labeled as facade, and the rest are labeled as slope. The simplification threshold ε for convex polygons during vertex simplification is set to 0.05 m~0.15 m.

4. The method for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub according to claim 3, characterized in that, In step S3, based on the obtained LOD2 shell model, the camera projection matrix output by the SfM algorithm is used to perform deep learning semantic segmentation on the facade image to obtain the opening mask. The mask corner points are mapped to adjacent views and triangulated using epipolar geometry to reconstruct the 3D opening corner points; Boolean difference culling is performed on the LOD2 mesh, opening geometry is inserted, and the LOD3 model with holes is output.

5. The method for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub according to claim 1, characterized in that, In step S4, depth testing, backface culling, and 4×MSAA are enabled on the browser side, and rotation, scaling, and roaming interactions are achieved through the trackball algorithm.

6. The method for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub according to claim 1, characterized in that, In step S5, the process of constructing the hybrid octagonal-binary-quadrilateral index is as follows: The root node is initialized with full-scene AABB; the first level uses an octree to divide the main buildings; the second level uses a binary tree to divide the layers along the elevation; the third level uses a quadtree to subdivide the equipment area in the plane; the leaf node stops when the side length or number of tiles reaches the threshold. Node information is written to the disk database in the structure of "node number-parent node number-level-bounding box-tile handle list" and stored for a long time. The browser can locate visible tiles in constant time by using the node number, reducing data input / output and transmission overhead.

7. The method for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub according to claim 1, characterized in that, In step S6, each rendered frame removes invisible branches by view frustum culling; The screen space error e is calculated in real time for visible tiles and compared with the threshold ε: when e>ε, a higher-level tile is incrementally loaded; when e<ε, it is retained or downgraded. Only LOD0 is loaded initially, and then LOD1-LOD3 are superimposed. When the tiles are very close, LOD3 is used to replace the lower-level tiles in the same area with LOD3 in a cross-fading manner, so as to achieve a smooth transition with addition as the main method and replacement as the auxiliary method, reduce the GPU rendering burden and maintain a 60 fps visual experience.

8. A device for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub, characterized in that, include: The image acquisition and attitude calculation module is used to acquire multi-view RGB images of the target port area, use the SfM algorithm for feature matching and epipolar constraint solving, obtain the camera attitude of each view and reconstruct sparse 3D point cloud. The geometric modeling module is used to generate a planar primitive point cloud based on the sparse 3D point cloud through Euclidean clustering and RANSAC plane fitting, and to construct a semantically labeled LOD2 shell model by orthogonally projecting the planar primitive point cloud and sequentially extracting convex polygons, simplifying vertices and triangulating it. The opening reconstruction and Boolean difference removal module is used to map the 2D openings obtained from deep learning semantic segmentation to 3D based on the camera projection matrix and epipolar geometry, and perform Boolean difference removal to automatically upgrade to a LOD3 model with holes; The formatting and transmission module is used to serialize LOD0-LOD3 models into glTF 2.0 binary format and transmit them to the client via HTTP / 2; The client-side rendering module is used to complete decompression, uploading, and PBR rendering in the WebGL 2.0 environment, enabling interactive, plugin-free visualization. The hybrid spatial index module is used to build an octagonal-binary-quadrilateral hybrid index, which performs multi-level partitioning and persistence of LOD0-LOD3 tiles, enabling clients to locate visible tiles in a constant time based on node numbers, reducing data input / output and transmission overhead. The progressive loading module is used to progressively load visible tiles using view frustum clipping and screen space error determination. It first loads LOD0, then incrementally overlays LOD1-LOD3, and replaces lower-level tiles in the same area with LOD3 using crossfading at very close distances, resulting in a smooth visual experience in the browser. "Very close distance" refers to the Euclidean distance d ≤ 100m from the center of the camera to the center of the target tile.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method for constructing and visualizing a multi-level digital twin simulation model of a port navigation and power hub as described in any one of claims 1 to 7.

Images