Indoor and outdoor integrated visualization method for 3D map building of smart city

Abstract

According to the smart city 3D map building indoor and outdoor integrated visualization method, based on a Unity rendering engine and a Mapbox map tool, an indoor and outdoor integrated 3D visualization model representation mode divided into a graph layer and a detail layer is researched and developed, 3D visualization is rapidly achieved, a vivid roaming effect is obtained, an environment similar to the real world is formed, and the method is suitable for popularization and application. Therefore, the user can carry out the dynamic interaction operation of the 3D scene. When outdoor roaming is carried out, only terrain, ground feature and building structure models in a visual range are rendered, and the internal structure and details of a building do not need to be loaded. When roaming into a room, a user can see the internal structure, characteristics, indoor real objects and outdoor scenes of the building at the current view angle. The indoor and outdoor integrated 3D visualization not only can carry out dynamic and vivid visual representation on the building on a computer to meet user requirements to the greatest extent, but also promotes research and application of indoor and outdoor integrated positioning and navigation, and promotes indoor intelligent management of the building and development of a digital city.

Description

Technical Field

[0001] The present application relates to a method for visualizing the indoor and outdoor integration of a three-dimensional map building, and in particular to a method for visualizing the indoor and outdoor integration of a 3D map building in a smart city, belonging to the technical field of 3D indoor and outdoor integration visualization. Background Art

[0002] With the rapid development of spatial information technology, high-resolution satellite imagery, high-speed network technology, large-capacity data processing and storage technology, computer science, computer graphics, visualization, and virtual reality technologies, 3D visualization has become a trend in information visualization. Much experience and methods have been accumulated in the fields of 3D maps and 3D architectural visualization. In recent years, public demand for integrated indoor and outdoor location-based services has intensified, and the related integrated indoor and outdoor 3D visualization has become a research hotspot in this field. Visualization of 3D scenes is a core component of building a "digital city." Urban 3D simulation and visualization systems can assist in urban planning, urban space management, and traffic route design. Urban 3D virtual reality can help firefighters and medical institutions conduct professional training, emergency drills, and emergency response. Navigation, location-based services, and virtual browsing applications based on visualization technology are bringing new lifestyles. Furthermore, virtual reality is increasingly being used in social entertainment.

[0003] Buildings are an integral part of the urban environment. Therefore, building a complete digital city application system requires integrating indoor applications within urban buildings. This is an inevitable trend in the future development of digital cities. With the development and widespread application of mobile internet and indoor positioning technologies, public demand for navigation applications has expanded from outdoor navigation to integrated indoor and outdoor navigation. Integrated indoor and outdoor 3D visualization is a key research topic.

[0004] Currently, data for both outdoor and indoor scenes is generated in the form of raster images. While relatively inexpensive and computationally fast, these images have significant limitations in terms of spatial cognition and are therefore unsuitable for roaming systems that require real-time imaging and large-scale scenes. 3D visualization methods based on graphical information are computationally intensive.

[0005] The irregular triangulated network for 3D scene data organization and visualization has high storage efficiency, simple data structure, low data redundancy, and high computational efficiency, but the algorithm is complex and difficult to implement. Regular grid data has high redundancy in areas with relatively flat terrain, and poor representation effect in areas with complex terrain or large terrain fluctuations.

[0006] The problems that need to be solved in existing 3D indoor and outdoor integrated visualization technologies and the key technical difficulties of this application include:

[0007] (1) Currently, data for both outdoor and indoor scenes are raster images, which have great limitations in terms of spatial cognition and are not suitable for roaming systems that require real-time imaging and large-scale scenes. The existing 3D visualization methods based on graphic information have a large amount of computation. The storage efficiency of irregular triangulated networks for 3D scene data organization and visualization is high, but the algorithm implementation is relatively complex and difficult. Regular grid data has high redundancy in areas with relatively flat terrain, and poor representation effect in areas with complex terrain or large terrain undulations. The existing technology lacks a 3D model representation method and data organization method that considers multiple levels of detail of the model based on indoor and outdoor 3D models. It lacks layered visualization technology based on Mapbox and relies on the Unity platform to implement a web virtual roaming system. The problem that building 3D modeling can only represent buildings as a single unit and lacks indoor and outdoor relationships has not been solved. The degree of precision of most 3DGIS models is low, and the problem of being limited to representing the surface texture of buildings cannot be solved. The inability to achieve indoor and outdoor integrated visualization of three-dimensional maps and buildings has restricted the application of urban 3D navigation, location services, and virtual browsing.

[0008] (2) The existing technology lacks the representation method of expanding 3D map models and the scene division of buildings, lacks the representation method of 3D map models and 3D building models suitable for integrated indoor and outdoor 3D visualization, and lacks building models that overlay fbx data; does not construct 3D building LOD models, lacks a key file system + relational database to store 3D building models; does not set up a data transmission process, lacks the transmission process from the server to the client and the process from client requesting data to rendering. The indoor and outdoor integrated rendering method has not been established, and dynamic rendering cannot be performed according to the visible area; lacks a visual roaming mechanism, and cannot enable users to switch from outdoor scenes to indoor scenes through user roaming; cannot realize the integrated indoor and outdoor visualization application of urban 3D map buildings; cannot perform dynamic and vivid visualization of buildings on computers, cannot participate in planning and design, cannot meet user requirements, and restricts the research and application of integrated indoor and outdoor positioning and navigation.

[0009] (3) The existing technology lacks the organization and expression of 3D models that integrate indoors and outdoors, lacks 3D map representation content and 3D map model representation methods, lacks outdoor and indoor representations of 3D building models, lacks multi-level of detail model representation and construction rules of 3D building LOD models; lacks 3D data organization methods based on Mapbox network map services, cannot access data based on Unity's http, lacks 3D visualization that integrates indoors and outdoors, cannot perform layered rendering that integrates indoors and outdoors, and cannot perform dynamic rendering based on the visible area; cannot switch between indoor and outdoor scenes through user roaming, and users cannot control the roaming position and direction according to their intentions. There are many problems with the integrated indoor and outdoor visualization of 3D maps and buildings that are not good for the user experience. Summary of the Invention

[0010] In response to the problems of the existing technology, this application is based on the Unity rendering engine and Mapbox map tools to develop a 3D visualization model representation method for indoor and outdoor integration with different layers and levels of detail, explore the visualization method of indoor and outdoor integration, quickly realize 3D visualization and obtain realistic roaming effects, and form an environment similar to the real world for users to perform dynamic interactive operations in 3D scenes. When roaming outdoors, only the terrain, landforms and building structure models within the visible range are rendered, and there is no need to load the internal structure and details of the building. When roaming into the room, you can see the internal structure, features, indoor objects and outdoor scenes of the building from the current perspective. The 3D visualization of indoor and outdoor integration can not only dynamically and vividly visualize the building on the computer, allowing users to immerse themselves in the scene and participate in planning and design, and meet user requirements to the greatest extent, but also promote the research and application of integrated indoor and outdoor positioning and navigation, and promote the development of intelligent indoor building management and the establishment of digital cities.

[0011] To achieve the above technical effects, the technical solutions adopted in this application are as follows:

[0012] A method for visualizing the indoor and outdoor integration of 3D map buildings in smart cities, expanding the representation of 3D map models and the scene division of buildings, and establishing a representation method for 3D map models and 3D building models suitable for integrated indoor and outdoor 3D visualization: the 3D map model adopts the representation method of DLG+grid base map+DEM, and overlays the building model of fbx data through the coordinate connection relationship between the building and the map; at the same time, a 3D building LOD model is constructed, the map data comes from the Mapbox map server, and a key file system+relational database is used to store the 3D building model; a data transmission process is set up, including the transmission process from the server to the client and the process from the client requesting data to rendering; an integrated indoor and outdoor rendering method is established: layered rendering is performed based on Mapbox, and dynamic rendering is performed according to the visible area; a roaming mechanism is implemented after visualization, and users can switch from outdoor scenes to indoor scenes through roaming; an integrated indoor and outdoor visualization application of urban 3D map buildings is realized;

[0013] 1) Indoor and outdoor integrated 3D model organization and expression: Construct 3D map representation content and 3D map model representation methods, taking into account the representation of the base map and the vectorized representation of the ground objects, and satisfying the representation of later interactive operations; construct the outdoor and indoor representations of the 3D building model, build the 3D building model based on the representation method, and then associate the 3D map model and building model through coordinate data based on the topological relationship between the two; construct a multi-level of detail model representation and design the construction rules of the 3D building LOD model;

[0014] 3D data organization method based on Mapbox web map service: for vector data, the vector slice data MVT structure is used to store model resource files on the server, and a relational database is used to store 3D building model information, which can be retrieved based on key-value queries;

[0015] HTTP data access based on Unity: Using desktop browser as 3D visualization client, WebGL as graphics rendering engine API, and Unity's communication class for network data transmission for web page network communication;

[0016] Dynamic rendering and caching design are adopted on the client to achieve rapid data acquisition and reduce the pressure of real-time rendering;

[0017] 2) Integrated indoor and outdoor 3D visualization: Indoor and outdoor layered rendering: Indoor and outdoor data are rendered in layers based on the Mapbox toolkit, and dynamic rendering is performed based on the visible area;

[0018] Interactive roaming: Based on Unity's roaming mechanism, indoor and outdoor scenes are switched through user roaming. The roaming mode adopts free roaming and interactive roaming. Users control the roaming position and direction according to their intentions.

[0019] Preferably, the integrated indoor and outdoor 3D model represents:

[0020] 1) 3D map model representation: A 3D map model is created using a combination of a digital line map (DLG) + a grid base map + a digital elevation model (DEM). This allows for integrated 3D visualization of both indoor and outdoor areas, with detailed representation of land features, especially buildings. The outdoor environment and landforms are also reflected to enable roaming.

[0021] 2) 3D building model representation: The building's exterior outline and interior objects together constitute the building as a whole. The 3D building model only represents the building's geometric structure and surface texture. The 3D building model representation is divided into exterior representation and interior representation. The integrated interior and exterior 3D building model includes both the building's exterior outline and interior objects.

[0022] 1) Outdoor Representation: 3D building models are built using geometric models. The outdoor representation is the building's outer contour, which consists of two parts: geometric structure and surface texture data. The model's geometric structure includes the building's main body, roof, and ancillary facilities. Based on the geometric structure, texture images are added to the building model.

[0023] 2) Indoor representation: including rooms, passages, connections, and indoor entities;

[0024] 3) Integrated indoor and outdoor representation: The outdoor 3D map model and 3D building model are displayed simultaneously. The 3D building model containing the building's outer contours and indoor scenes is placed in the outdoor 3D map model according to its actual geographic coordinates in reality. Through the integrated indoor and outdoor representation, both the indoor 3D scene and the outdoor surrounding environment scene can be observed.

[0025] Preferably, the data organization is integrated indoors and outdoors:

[0026] 1) Ground Data Organization: 3D map models are implemented using a combination of a digital line map (DLG) + a raster basemap + a digital elevation model (DEM). The DEM data comes from raster data, and the raster basemap uses raster slice data. When the client requests raster slices, the server renders them using Mapnik. The DLG representing ground features uses vector slice data, sourced from MVT data in the Mapbox map service.

[0027] The terrain raster data format is Mapbox's Terrain-RGB. The data contains global elevation data encoded in raster slices. The RGB color values ​​in the slices are decoded into the original height in meters. A DEM is constructed based on the elevation data. The DEM data is organized and represented in a regular rectangular grid. Regular points are collected on the slices to form a 3D terrain model of the regular rectangular grid. The surface raster slice data is then overlaid on the 3D terrain model.

[0028] The DLG representing the ground features uses the street vector slice data MVT. The vector slices are organized by layers. Each layer is named according to the ground feature it corresponds to. It contains two parts: geometric features and metadata information. The slices contain layers, and each layer stores the corresponding ground features.

[0029] Each vector tile stores at least one layer, and each layer stores at least one feature. Each feature encodes geometry and feature attributes in different ways. Vector tiles convert geographic coordinates into vector tile grid coordinates. The origin of the coordinate system is located in the upper left corner of the tile, with the X axis pointing rightward and the Y axis pointing downward. Points, lines, and polygons are encoded as x / y pairs. The integer pairs in the tag field correspond to feature attributes. In each tag pair, there are key and value pairs, respectively. The first integer is the index number of the key in the keys list of the layer to which it belongs. Similar to the first integer, the second integer is the index number of the value in the values ​​list. The index of each feature key is unique, ensuring that each feature does not have duplicate attributes.

[0030] 2) Data Organization of 3D Building Models 3D models use the FBX format and a SceneGraph / Tree structure to store all model information. The root node of the file scene tree contains geometric networks, cameras, light sources, and skeleton nodes, and the corresponding attribute information is read by traversing;

[0031] The integrated indoor and outdoor 3D building model is divided into the outer contour model and the indoor object model. The 3D building model is constructed with LOD. The representation of a building will have a set of 3D building models of different precisions, each of which contains one or more FBX files.

[0032] The building is divided into the building outline and interior space, where the building outline includes the roof and exterior walls; the interior space is organized according to the divided elements; the 3D model data of the interior space is organized according to the topological relationship of the interior space.

[0033] Preferably, the data transmission process from server to client is as follows: map data is pre-generated into a data pyramid and stored in blocks on the server. The user obtains the map slice of the specified scale and location by sending a network request with parameters to the server, where the URL and the resource correspond one to one;

[0034] The workflow for requesting Mapbox's tile map service is as follows:

[0035] (1) The client sends a request: The client sends an HTTP request. The requested tile URL format is http: / / ip:port / version / {map_id} / {z} / {x} / {y}.filenameSuffix?param=value; where version is the version of the current map data; filenameSuffix is ​​the file suffix, i.e. the format of the request data; {z}{x}{y} are tile map parameters, {z} is the scale of the tile map, and {x}{y} is the index value of the tile. The tile map parameters are calculated based on the geographic coordinate value and the display scale.

[0036] (2) The remote server parses the path to obtain the coordinates and extension suffix: if there is a cache, read the slice in the cache; if there is no data in the cache, obtain the specified map slice according to the URL format;

[0037] (3) Execute the query: Calculate the range based on the coordinates and query the data to generate tiles; a) First connect to the database; b) Execute the query statement and write the query results to the newly created vector tile; If the request is for a raster tile, render the vector tile into a raster tile through the API; c) Close the database; d) Save the newly created tile and keep the tile in the cache database; e) Send the data to the client.

[0038] Preferably, the process from client request to rendering is the process from client interaction with the map to map rendering update. If the intermediate data source changes or the visible tile set changes, the changed tiles need to be obtained and decoded and passed into the buffer for rendering;

[0039] (1) Event triggering and tile updating: The client process from request to rendering starts when the user initiates a request event. When the map is initialized, Clear is first used to clear all currently activated tiles, and then OnInitialized and OnTileAdded of the tiles are triggered to add the tiles at the current location. When the map is updated, OnTileAdded and OnTileRemoved are triggered to add tiles within the current visible area and remove tiles outside the visible area. When the map is zoomed or moved, the tile set within the visible area is recalculated, and the updateMap() method of the Map class is called to update the map.

[0040] (2) Requesting tile resources: The tile set maintains the tiles within the current visible range. After the update is successful, the raster and vector resources on the tile are requested through the IFileSource interface according to the tile CanonicalTileld. The request is an HTTP request. After requesting the server, the resources are returned and the layer is initialized. When sending an HTTP request in Unity to obtain data on the map server, the UnityWebRequest library class is used. The request method consists of three elements.

[0041] (3) Tile decoding: The data transmitted from the server is serialized binary format data encoded in the vector tile transmission format. The tile data stored in the local cache is also intact binary format data. The original data is decoded and used to build the model. VectorTile calls VectorTileReader to read the tile data for decoding, i.e. serialization, and processes the stored attribute information and coordinate information. A tile contains multiple geographic feature layers, and the layers are read and decoded in a loop, and the features in the layers are decoded one by one. Then the decoded data is visualized in layers.

[0042] Preferably, based on Mapbox's layered rendering: the layers are visualized in layers, and the layers are divided into surface layers and building layers, wherein the surface layer is represented by a 3D terrain model consisting of a terrain network and surface textures, and the feature layer is divided into an indoor and outdoor integrated building layer represented by a 3D building model, and a road layer, a general building layer, and a land cover layer visualized by MVT data;

[0043] Use the following scripting interface for layer visualization:

[0044] (1) MapVisualizer: implements map visualization, creates requested map slices and relays them to the required Factory, reuses slices based on a caching mechanism, and locates slices in the Unity world;

[0045] (2) Factory: Processes the map's terrain, images, and vector data for rendering;

[0046] (3) LayerVisualizer: receives vector data and creates and styles features (such as buildings) as layers on the map;

[0047] (4) Stacks and Modifiers: Stacks are a collection of Modifiers. Modifiers are scripts that create, modify, and style features.

[0048] This implementation of MapVisualizer requires setting up several Factories:

[0049] (1) TerrainFactory, requests and creates the world base grid, i.e., the terrain grid;

[0050] (2) MapImageFactory, requests map raster images and provides materials and textures to TerrainFactory;

[0051] (3) VectorTileFactory, which sets up a relatively complex interface in the Factory that processes vector data;

[0052] In VectorTileFactory, you can request one or more of the following vector data types: streets, terrain, and traffic. The request will merge the datasets and return a data package. Each data source is visualized as a layer. The traffic dataset layer uses lines of different colors to represent traffic flow conditions. The terrain dataset layer has three source layers: a terrain overlay layer, a hillside layer, and a highline layer. The street dataset has the largest amount of data and consists of various urban elements. Multiple data sources are visualized using one layer. To maintain the orderliness and intuitiveness of the data organization, each data source is visualized on its own layer.

[0053] The layer visualizer distributes the received vector data to the modifier stack for processing by layer. The modifier stack contains two types of modifiers, game object modifiers and mesh modifiers, which are used to create and decorate game objects. The mesh modifier generates the data required for the game object mesh. First, all mesh modifiers are run to generate all the data required for the mesh, and the mesh data is used to create the game object. Then all game object modifiers are run to decorate the game object.

[0054] Preferably, the indoor and outdoor integrated rendering process: after the data is processed by the CPU and GPU from the memory, the image is displayed using the video memory;

[0055] Rendering of 3D scenes is based on a programmable pipeline. The process is to draw 3D objects on a 2D screen. The visualization result is to convert the mathematical vertices into pictures, and the user sees a series of 2D images.

[0056] Convert 3D objects into 2D images and transfer data from the CPU to the GPU. The data transferred includes 3D object information, viewpoint information, light source information, lighting model, and texture data. Finally, a 2D image is rendered. The rendering pipeline is divided into three stages.

[0057] 1) Application stage: Data is transferred from memory to the CPU for calculation. The memory and CPU transmit the data generated after a series of calculations in Unity, such as frustum clipping, scene graph creation, and collision detection, to the GPU through the data bus for processing in the next stage. The generated vertex coordinates and texture coordinates are used in the next vertex processing stage, and the texture information is used in the rasterization stage.

[0058] 2) Geometry phase: This is performed on the GPU and is responsible for 3D vertex coordinate transformation and lighting calculation. During this phase, vertex coordinate transformation, screen clipping, lighting attribute processing, projection transformation, and screen mapping are performed in sequence. Finally, the transformed and projected vertex coordinates, texture coordinates, and colors are calculated.

[0059] Get the screen coordinates of the object's vertices, and then use the original connection relationship between the vertices to calculate the mesh structure of the object. The index and vertex are the two major components of the mesh. The latter is linked to the former to form the basic geometric unit, which is then clipped to retain only the content within the screen range. The converted vertex coordinates and fragments required for the next stage of drawing are output;

[0060] 3) Rasterization stage: The color and depth values ​​corresponding to each pixel in the fragment are calculated using the texture information generated by the application stage. The color value of the pixel will be written and saved in the frame buffer;

[0061] The steps of the rasterization stage are: rasterizing and interpolating the fragment and vertex coordinates, and then performing pixel operations and pixel frame buffering; the 3D model is projected onto a two-dimensional plane through vertex transformation and assembled into primitives.

[0062] Preferably, indoor and outdoor integrated roaming:

[0063] 1) Interactive roaming: Users use character controllers to interactively roam in a 3D virtual environment, observing the virtual scene from the character controller's perspective. During the roaming process, they analyze the scene and calculate the path. By continuously moving the viewpoint or changing the direction of sight, and using the mouse and keyboard as system input devices, users can interact with the virtual scene.

[0064] 2) Indoor and outdoor scene switching: On the one hand, users can operate through free roaming to observe indoor and outdoor scenes from any angle; or use character roaming to control the character controller to walk outdoors to observe the surrounding environment and enter indoor scenes for observation. On the other hand, when switching from outdoor to indoor, the level of detail of indoor and outdoor building objects is achieved by controlling the resolution. The level of detail of the building model changes from coarse to precise, and the visibility of the building objects changes from non-existent to present. Indoor and outdoor objects change from visible to invisible according to the percentage of the building model in the view.

[0065] In indoor scenes, the transparency of the building's interior walls is controlled based on the character controller's position. When the character controller reaches a certain floor of the building, the walls of that floor are set to semi-transparent, allowing the character controller to intuitively see the building's interior structure.

[0066] 3) Collision detection mechanism: This mechanism uses mathematical calculations to determine whether two non-penetrating objects in a spatial range intersect. Collision detection in this scenario is converted into an intersection problem of geometric models.

[0067] The bounding sphere is the smallest sphere that contains the object. By directly detecting whether the distance between the center of the circle is less than the radius, we can detect whether the objects intersect.

[0068] In 3D space, AABB is a box-shaped cuboid with the normal of each face parallel to a given coordinate axis. Only the relationship between the corresponding coordinate axes is compared to detect whether there is an intersection. OBB is the smallest cuboid that contains an object and is relative to any coordinate axis direction. It has directionality. If the projections of two OBBs on all projection axes overlap, it is determined to be a collision; otherwise, it is determined that no collision has occurred.

[0069] Preferably, the 3D map building indoor and outdoor integrated visualization system consists of:

[0070] 1) System Architecture: Developed based on the Unity engine and published on a web platform, it adopts a browser / server model B / S structure and consists of three parts: the presentation layer, the business logic layer, and the data layer, with the core part being the presentation layer.

[0071] The data layer includes the raster map database and vector data database provided by the Mapbox map server; the model relational database that stores model properties and model file paths; and the model resource files of the 3D building model;

[0072] The business logic layer receives client requests, such as requests for sliced ​​data of a specified scale in a specified area, parses the request parameters, initiates a query to the database, and finally encapsulates the query results and returns them to the presentation layer.

[0073] The presentation layer responds to various user operations, parses and renders the 3D building model data, 3D map model data, and path data returned by the control layer, and realizes visualization and interactive map operations: character roaming and path query;

[0074] II) Overall process of indoor and outdoor integrated visualization system

[0075] Step 1: Get the map area based on the current visible area range;

[0076] Step 2: The web page sends a request to obtain vector slices and Ou Ge slice maps. The remote server requests the database to obtain raster images and binary files encoded by the vector slices, and returns the data to the web page. The web page decodes the received binary slice files and then renders the 3D map layer. While requesting the map data, the database is requested to query the 3D building models within the current map range and request the model resource files.

[0077] Step 3: Build a 3D map model and render it in layers;

[0078] Step 4: Load the 3D building LOD model in the visible area by layer;

[0079] Step 5: If the user's operation causes the visible area to change at this time, return to the first step to request and render again. If the visible area does not change, this process ends, realizing indoor and outdoor integrated 3D visualization.

[0080] Preferably, the roaming module has three submodules: free roaming, character roaming and small map; the system increases user immersion, and in addition to character-free roaming, there are first-person and third-person roaming;

[0081] Switching between free roaming and character roaming means switching the camera, and controlling the corresponding camera SetActive property to false or true;

[0082] 1) Free roaming: Users can observe the 3D scene from any angle by operating the camera themselves. They can control the rotation of the camera by right clicking the mouse, and the left, right, and forward movement of the camera by using the WSAD keys on the keyboard, that is, the movement of the viewing angle. The latitude and longitude of the intersection of the window position ray (a ray that passes through the viewpoint from the camera) and the map plane are determined based on the current camera position. The map slice code facing the current camera can be obtained. According to the set slice visual range, the slice data of the corresponding slice number is requested. The requested slice number will be marked as active. After the map is operated and the camera is moved, the map slice range will change. At this time, if the requested slice is active, that is, the existing slice does not need to be requested again. If the value of the slice and the center slice exceeds the destruction range value, it does not need to be placed in memory and will be destroyed immediately.

[0083] 2) Character roaming:

[0084] Step 1: Add a terrain collider to the ground;

[0085] Step 2: Add a first-person controller. First, create an empty GameObject called FPSController. Then, add a rigidbody control under this GameObject to give it gravity so that it is controlled by gravity. Next, add a character control script so that the user can control the object's movement, forward and backward, left and right, as well as walk and run, using the mouse and keyboard. Finally, add a new GameObject called Camera under the FPSController object to act as the user's eyes and follow the controller's movements.

[0086] Step 3: Add a third-person controller to see the mission model from a third-person perspective. Add a mission model under the controller, add a rigid body and a collider to achieve a real physical world effect. At the same time, add a character control script and a camera follow script. When the character model moves under user control, the camera will continue to follow the character, ensuring that the camera is always behind the character to shoot. Finally, place the main camera in the third-person controller and bind the camera and character together.

[0087] Step 4: Add a perspective switching button to the system interface. When switching perspectives, not only the character controller is switched, but also the map camera needs to be replaced, allowing users to observe the scene from different perspectives.

[0088] 3) Minimap: Displays the surrounding environment of the roaming protagonist, with the roaming protagonist as the center. This is achieved by removing the main camera, adding an additional camera, and setting this camera to perspective projection. Perspective projection increases the projection area, and finally displays the content of the projection area on the UI control;

[0089] First, add a new camera to the scene and name it to distinguish it from the main camera. Make sure the roaming character is always in the center of the mini-map. Set the camera as a child object of the roaming character and set its coordinates above the character model. Aim the camera at the character model. The image captured by the camera will be the image in the mini-map. Then display the camera image on the created Canvas render texture. At this point, the camera image is displayed as a control on the interface.

[0090] To make the roaming character more conspicuous in the minimap, different layers need to be set. First, add a model such as a sphere to the roaming character. The main camera shoots the character model, while the character model is displayed as a sphere in the minimap. Add two layers, a minimap layer MinimapLayer and the other is the character model layer PersonLayer. Then set the layer of the sphere to the newly added layer MinimapLayer; set the roaming character model to the PersonLayer layer, and finally modify the corresponding camera rendering layer. The main camera renders the PersonLayer layer instead of the MinimapLayer layer, while the minimap camera does the opposite.

[0091] Compared with the existing technology, the innovation and advantages of this application are:

[0092] (1) In response to the problems of the existing technology, this application develops a 3D visualization model representation method for indoor and outdoor integration based on the Unity rendering engine and Mapbox map tools, which is divided into layers and detailed levels. It explores the visualization method of indoor and outdoor integration, quickly realizes 3D visualization and obtains realistic roaming effects, and forms an environment similar to the real world for users to perform dynamic interactive operations in 3D scenes. When roaming outdoors, only the terrain, landforms and building structure models within the visible range are rendered, and there is no need to load the internal structure and details of the building. When roaming into the room, the internal structure, features, indoor objects and outdoor scenes of the building from the current perspective can be seen. The 3D visualization of indoor and outdoor integration can not only dynamically and vividly visualize the building on the computer, allowing users to immerse themselves in the scene and participate in planning and design, and meet user requirements to the greatest extent, but also promote the research and application of integrated indoor and outdoor positioning and navigation, and promote the development of intelligent indoor building management and the establishment of digital cities.

[0093] (2) This application expands the representation of 3D map models and the scene division of buildings, establishes a representation of 3D map models and 3D building models suitable for integrated indoor and outdoor 3D visualization, and superimposes the building model of fbx data through the coordinate connection relationship between the building and the map; at the same time, a 3D building LOD model is constructed, and a key file system + relational database is used to store the 3D building model; a rendering method for integrated indoor and outdoor is established, layered rendering is performed based on Mapbox, and dynamic rendering is performed according to the visible area; a roaming mechanism is implemented after visualization, and users can switch from outdoor scenes to indoor scenes through roaming; and an integrated indoor and outdoor visualization application of urban 3D map buildings is realized; based on the indoor and outdoor 3D models, this application considers the multiple levels of detail of the model, designs an integrated indoor and outdoor 3D model representation method and data organization method, develops a layered visualization technology based on Mapbox, and relies on the Unity platform to realize a web virtual roaming system. This solves the problem that 3D building modeling can only represent buildings individually and lacks indoor and outdoor relationships, can improve the level of refinement in most 3D GIS modeling, and can also solve the problem of being limited to representing building surface textures.

[0094] (3) This application proposes a representation method for 3D map models and 3D building models suitable for integrated indoor and outdoor 3D visualization, constructs 3D map models and integrated indoor and outdoor building 3D models, explores a technology for integrated indoor and outdoor 3D visualization and rendering that combines Mapbox and Unity through practice, develops a design program based on the Unity development engine, and implements an application example of integrated indoor and outdoor 3D visualization. This application expands the representation method of traditional 3D map models and the scene division of buildings, proposes a representation method for 3D map models and 3D building models suitable for integrated indoor and outdoor 3D visualization, describes the data transmission process, proposes an integrated indoor and outdoor rendering method, and establishes a mechanism for roaming after visualization. Through user roaming, users can switch from outdoor scenes to indoor scenes. The application example shows the visualization effects of indoor and outdoor scenes, the integrated display effect of entering indoors from outdoor, the user roaming effect, and the query effect. It can help firefighting and medical institutions conduct professional training, plan exercises, and emergency command; navigation, location services, and virtual browsing applications based on visualization technology have brought new lifestyles; at the same time, the method of this application is increasingly used to implement virtual reality technology in social entertainment. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 It is a schematic diagram of a 3D terrain model of a regular rectangular grid.

[0096] Figure 2 It is a schematic diagram of the topological relationship of indoor space.

[0097] Figure 3It is a diagram of the hierarchical structure of visualization layers.

[0098] Figure 4 This is a schematic diagram of the processing flow of 3D data in the GPU.

[0099] Figure 5 It is a diagram showing the percentage of the view occupied by the object in the picture and the model hierarchy relationship.

[0100] Figure 6 It is a 3D map building indoor and outdoor integrated visualization system architecture diagram.

[0101] Figure 7 It is the overall flow chart of the indoor and outdoor integrated visualization system.

[0102] Figure 8 It is a diagram of the module composition of the 3D map building indoor and outdoor integrated visualization system.

[0103] Figure 9 This is a flowchart for creating a character walkthrough.

[0104] Figure 10 This is a schematic diagram of the experimentally constructed 3D building model.

[0105] Figure 11 It is a hierarchical diagram of the building model at different viewing distances. DETAILED DESCRIPTION

[0106] Below, in conjunction with the accompanying drawings, the technical solution of the smart city 3D map building indoor and outdoor integrated visualization method provided by this application is further described so that technical personnel in this field can better understand this application and implement it.

[0107] This application explores a method for integrated indoor and outdoor 3D visualization. Based on the existing representation methods of 3D map models and 3D building models, a 3D map model and building model suitable for integrated indoor and outdoor visualization are selected. Mapbox is used to implement layered rendering of model data, and Unity is used to achieve integrated indoor and outdoor interactive roaming. Finally, the method of this application is verified by implementing a prototype system.

[0108] 1) Indoor and outdoor integrated 3D model organization and expression: Construct 3D map representation content and 3D map model representation methods, taking into account the representation of the base map and the vectorized representation of the ground objects, and satisfying the representation of later interactive operations; construct the outdoor and indoor representations of the 3D building model, build the 3D building model based on the representation method, and then associate the 3D map model and building model through coordinate data based on the topological relationship between the two; construct a multi-level of detail model representation and design the construction rules of the 3D building LOD model;

[0109] 3D data organization method based on Mapbox web map service: for vector data, the vector slice data MVT structure is used to store model resource files on the server, and a relational database is used to store 3D building model information, which can be retrieved based on key-value queries;

[0110] HTTP data access based on Unity: Using desktop browser as 3D visualization client, WebGL as graphics rendering engine API, and Unity's communication class for network data transmission for web page network communication;

[0111] Dynamic rendering and caching design are adopted on the client to achieve rapid data acquisition and reduce the pressure of real-time rendering;

[0112] 2) Integrated indoor and outdoor 3D visualization: Indoor and outdoor layered rendering: Indoor and outdoor data are rendered in layers based on the Mapbox toolkit, and dynamic rendering is performed based on the visible area;

[0113] Interactive roaming: Based on Unity's roaming mechanism, indoor and outdoor scenes are switched through user roaming. The roaming mode adopts free roaming and interactive roaming. Users control the roaming position and direction according to their intentions.

[0114] 1. Indoor and outdoor integrated 3D model organization expression

[0115] 3D model organization associates outdoor scenes with indoor scenes, and establishes an integrated indoor and outdoor 3D model representation through outdoor 3D model representation and indoor 3D architectural model representation.

[0116] (1) 3D model representation of indoor and outdoor integration

[0117] 1) 3D map model representation

[0118] The 3D map model is realized by using the representation method of digital line map DLG + grid base map + digital elevation model DEM, achieving 3D visualization of indoor and outdoor integration, setting detailed representation of land features, especially buildings; at the same time, reflecting the outdoor environment and landforms to realize roaming.

[0119] 2) 3D building model representation

[0120] The outdoor outline and indoor objects of a building together constitute the entire building. The 3D building model only represents the building's geometric structure and surface texture. The 3D building model representation is divided into outdoor representation and indoor representation. The indoor and outdoor integrated 3D building model includes two parts: the building's outdoor outline and indoor objects.

[0121] 1) Outdoor Representation: 3D building models are built using geometric models. The outdoor representation is the building's outer contour, which consists of two parts: geometric structure and surface texture data. The model's geometric structure includes the building's main body, roof, and ancillary facilities. Based on the geometric structure, texture images are added to the building model.

[0122] 2) Indoor representation: including rooms, passages, connections, and indoor entities;

[0123] 3) Indoor and outdoor integration

[0124] The outdoor 3D map model and 3D building model are represented at the same time. The 3D building model including the building outline and indoor scene is placed in the outdoor 3D map model according to its actual geographic coordinates in reality. Through the integrated representation of indoor and outdoor, both the indoor 3D scene and the outdoor surrounding environment scene can be observed.

[0125] (2) Integrated indoor and outdoor data organization

[0126] 1) Ground data organization

[0127] The 3D map model is realized by using the representation method of digital line map DLG + raster base map + digital elevation model DEM. The DEM data comes from raster data, and the raster base map uses raster slice data. When the client requests raster slices, the server uses Mapnik to render vector slices. The vector slice data used by DLG to represent the ground features comes from the MVT data in the Mapbox map service.

[0128] Terrain raster data is in Mapbox's Terrain-RGB format. Terrain-RGB data contains global elevation data encoded in raster slices. The color values ​​in the slices are decoded into the original height in meters. A DEM is constructed based on the elevation data. The DEM data is organized and represented in a regular rectangular grid. Regular points are collected on the slices to form a DEM. Figure 1 A 3D terrain model of a regular rectangular grid is created. Surface grid slice data is then overlaid on the 3D terrain model.

[0129] The DLG representing the land features adopts the street vector slice data MVT. The vector slice organizes the data by layers. Each layer is named according to its corresponding land feature, which contains two parts: geometric features and metadata information. The slice contains layers, and each layer stores the corresponding land feature.

[0130] Each vector tile stores at least one layer, and each layer stores at least one feature. Each feature encodes geometry and feature attributes in a different way. Vector tiles convert geographic coordinates into vector tile grid coordinates. The origin of this coordinate system is located in the upper left corner of the tile, with the X-axis pointing right and the Y-axis pointing downward. Points, lines, and polygons are encoded as x / y pairs. The integer pairs in the tag field correspond to feature attributes. In each tag pair, there is a key and a value. The first integer is the index number of the key in the keys list of the layer to which it belongs. Similar to the first integer, the second integer is the index number of the value in the values ​​list. The index of each feature key is unique, ensuring that each feature does not have duplicate attributes. When the geometric data is large, this encoding method eliminates redundancy caused by attributes with the same key and similar values.

[0131] 2) Data Organization of 3D Building Models

[0132] The 3D model uses the FBX format and stores all the model information in a SceneGraph / Tree structure. The root node of the file scene tree contains the geometric network, camera, light source, and skeleton nodes, and the corresponding attribute information is read by traversing.

[0133] The integrated indoor and outdoor 3D building model is divided into the outer contour model and the indoor object model. The 3D building model is constructed with LOD. The representation of a building will have a set of 3D building models of different precisions, each of which contains one or more FBX files.

[0134] The building is divided into the building outline and the interior space, where the building outline includes the roof and the exterior wall; the interior space is organized according to the divided elements; the 3D model data of the interior space is organized according to the interior space topological relationship (see Figure 2 ) to organize.

[0135] 3) Indoor and outdoor integrated data organization

[0136] The spatial topological relationship between the 3D building model and the outdoor 3D terrain model is an overlay relationship. The 3D building model and the outdoor 3D map model are organized in different layers. The two layers are overlaid according to the geographical location of the building model. The indoor and outdoor models are connected through unified coordinates to achieve indoor and outdoor integration.

[0137] A hybrid file + database management method is used for 3D building model data. 3D model data is stored in the file system, and the attribute data of the 3D model is stored in the MySQL relational database. Model data is stored in files.

[0138] The outdoor 3D map model uses Mapbox data. Mapbox data consistently uses the Web Mercator projection. The coordinate attributes of the 3D building model are projected using the Web Mercator projection to achieve coordinate unification.

[0139] 2. 3D visualization of indoor and outdoor integration

[0140] (1) Data transmission process

[0141] 1) Server to client data transmission process

[0142] Map data is pre-generated into a data pyramid and stored in blocks on the server. Users can obtain map slices of a specified scale and location by sending a network request with parameters to the server. The URL and resource have a one-to-one correspondence.

[0143] The workflow for requesting Mapbox's tile map service is as follows:

[0144] (1) The client sends a request: The client sends an HTTP request. The requested tile URL format is http: / / ip:port / version / {map_id} / {z} / {x} / {y}.filenameSuffix?param=value; where version is the version of the current map data; filenameSuffix is ​​the file suffix, i.e. the format of the request data; {z}{x}{y} are tile map parameters, {z} is the scale of the tile map, and {x}{y} is the index value of the tile. The tile map parameters are calculated based on the geographic coordinate value and the display scale.

[0145] (2) The remote server parses the path to obtain the coordinates and extension suffix: if there is a cache, read the slice in the cache; if there is no data in the cache, obtain the specified map slice according to the URL format;

[0146] (3) Execute the query: Calculate the range based on the coordinates and query the data to generate tiles; a) First connect to the database; b) Execute the query statement and write the query results to the newly created vector tile; If the request is for a raster tile, render the vector tile into a raster tile through the API; c) Close the database; d) Save the newly created tile and keep the tile in the cache database; e) Send the data to the client.

[0147] 2) Client rendering process

[0148] The process from client request to rendering is the process from client interaction with the map to map rendering update. If the intermediate data source changes or the visible tile set changes, the changed tiles need to be obtained and decoded and passed into the buffer for rendering.

[0149] (1) Event triggering and tile updating: The client process from request to rendering starts when the user initiates a request event. When the map is initialized, Clear is first used to clear all currently activated tiles, and then OnInitialized and OnTileAdded of the tiles are triggered to add the tiles at the current location. When the map is updated, OnTileAdded and OnTileRemoved are triggered to add tiles within the current visible area and remove tiles outside the visible area. When the map is zoomed or moved, the tile set within the visible area is recalculated, and the updateMap() method of the Map class is called to update the map.

[0150] (2) Requesting tile resources: The tile set maintains the tiles within the current visible range. After the update is successful, the raster and vector resources on the tile are requested through the IFileSource interface according to the tile CanonicalTileld. The request is an HTTP request. After requesting the server, the resources are returned and the layer is initialized. When sending an HTTP request in Unity to obtain data on the map server, the UnityWebRequest library class is used. The request method consists of three elements.

[0151] (3) Tile decoding: The data transmitted from the server is serialized binary format data encoded in the vector tile transmission format. The tile data stored in the local cache is also intact binary format data. The original data is decoded and used to build the model. VectorTile calls VectorTileReader to read the tile data for decoding, i.e. serialization, and processes the stored attribute information and coordinate information. A tile contains multiple geographic feature layers, and the layers are read and decoded in a loop, and the features in the layers are decoded one by one. Then the decoded data is visualized in layers.

[0152] (2) Indoor and outdoor integrated rendering

[0153] 1) Layered rendering based on Mapbox

[0154] according to Figure 3 The structure shown visualizes layers in layers. Layers are divided into surface layers and building layers. The surface layer is represented by a 3D terrain model consisting of a terrain network and surface textures. The feature layers are divided into indoor and outdoor integrated building layers represented by 3D building models, and road layers, general building layers, and land cover layers visualized using MVT data.

[0155] Use the following scripting interface for layer visualization:

[0156] (1) MapVisualizer: implements map visualization, creates requested map slices and relays them to the required Factory, reuses slices based on a caching mechanism, and locates slices in the Unity world;

[0157] (2) Factory: Processes the map's terrain, images, and vector data for rendering;

[0158] (3) LayerVisualizer: receives vector data and creates and styles features (such as buildings) as layers on the map;

[0159] (4) Stacks and Modifiers: Stacks are a collection of Modifiers. Modifiers are scripts that create, modify, and style features.

[0160] This implementation of MapVisualizer requires setting up several Factories:

[0161] (1) TerrainFactory, requests and creates the world base grid, i.e., the terrain grid;

[0162] (2) MapImageFactory, requests map raster images and provides materials and textures to TerrainFactory;

[0163] (3) VectorTileFactory, which sets up a relatively complex interface in the Factory that processes vector data;

[0164] In VectorTileFactory, you can request one or more of the following vector data types: streets, terrain, and traffic. The request will merge the datasets and return a single data package. Each data source is visualized as a layer. The traffic dataset layer uses lines of different colors to represent traffic flow conditions. The terrain dataset layer has three source layers: a terrain overlay layer, a hillside layer, and a highline layer. The street dataset has the largest amount of data and is composed of various urban elements. Multiple data sources are visualized using one layer. To maintain the orderliness and intuitiveness of the data organization, each data source is visualized on its own layer.

[0165] The layer visualizer distributes the received vector data to the modifier stack for processing by layer. The modifier stack contains two types of modifiers, game object modifiers and mesh modifiers, which are used to create and decorate game objects. The mesh modifier generates the data required for the game object mesh. First, all mesh modifiers are run to generate all the data required for the mesh, and the mesh data is used to create the game object. Then all game object modifiers are run to decorate the game object.

[0166] 2) Rendering process

[0167] After the data is processed by the CPU and GPU from the memory, the image is displayed using the video memory.

[0168] Rendering of 3D scenes is based on a programmable pipeline. The process is to draw 3D objects on a 2D screen. The visualization result is to convert the mathematical vertices into pictures, and the user sees a series of 2D images.

[0169] Convert 3D objects into 2D images and transfer data from the CPU to the GPU. The data transferred includes 3D object information, viewpoint information, light source information, lighting model, and texture data. Finally, a 2D image is rendered. The rendering pipeline is divided into three stages.

[0170] 1) Application Stage: Data is transferred from memory to the CPU for computation. The memory and CPU transfer the data generated by Unity's frustum scissoring, scene graph construction, and collision detection calculations to the GPU via the data bus for processing in the next stage. The generated vertex coordinates and texture coordinates are used in the next vertex processing stage, and texture information is used in the rasterization stage. Figure 4 Shows the data processing flow in the GPU.

[0171] 2) Geometry phase: This is performed on the GPU and is responsible for 3D vertex coordinate transformation and lighting calculation. During this phase, vertex coordinate transformation, screen clipping, lighting attribute processing, projection transformation, and screen mapping are performed in sequence. Finally, the transformed and projected vertex coordinates, texture coordinates, and colors are calculated.

[0172] Get the screen coordinates of the object's vertices, and then use the original connection relationship between the vertices to calculate the mesh structure of the object. The index and vertex are the two major components of the mesh. The latter is linked to the former to form the basic geometric unit, which is then clipped to retain only the content within the screen range. The converted vertex coordinates and fragments required for the next stage of drawing are output;

[0173] 3) Rasterization stage: The color and depth values ​​corresponding to each pixel in the fragment are calculated using the texture information generated by the application stage. The color value of the pixel will be written and saved in the frame buffer;

[0174] The steps of the rasterization stage are: rasterizing and interpolating the fragment and vertex coordinates, and then performing pixel operations and pixel frame buffering; the 3D model is projected onto a two-dimensional plane through vertex transformation and assembled into primitives.

[0175] 3) Dynamic rendering of the visible area

[0176] Dynamic rendering is used. Only when the user operates the map will the program request the slice data in the visible area; otherwise, it remains in a waiting state until the user operates the map.

[0177] Tiles at different zoom levels based on the Web Mercator projection tile quadtree structure and the QuadTreeTileProvider script are used to request tiles within the visible area. The latitude, longitude, camera viewport, and map zoom level are used to determine the map range within the visible area. The map is also updated when the user moves, and the basemap layer and building layer are also updated.

[0178] (3) Indoor and outdoor integrated roaming

[0179] 1) Interactive Tour

[0180] Users use character controllers to interactively roam in a 3D virtual environment, observe virtual scenes from the perspective of the character controller, analyze scenes and calculate paths during roaming, and interact with virtual scenes by continuously moving the viewpoint or changing the direction of sight, using the mouse and keyboard as input devices for the system.

[0181] 2) Indoor and outdoor scene switching

[0182] On the one hand, users can operate through free roaming to observe indoor and outdoor scenes from any angle; or use character roaming to control the character controller to walk outdoors to observe the surrounding environment and enter indoor scenes for observation. On the other hand, in the process of moving from outdoor to indoor, the level of detail of indoor and outdoor building objects is achieved by controlling the resolution. The level of detail of the building model changes from coarse to precise, and the visibility of the building's internal objects changes from non-existent to visible. Indoor and outdoor objects change from visible to invisible according to the percentage of the building model in the view.

[0183] Figure 5 This shows the relationship between the percentage of the object in the frame and the model hierarchy. Model accuracy is set based on the distance from the camera. Models closer to the camera have higher accuracy settings, resulting in high-precision models, while models farther from the camera have lower accuracy settings, resulting in low-precision models. High-poly models and low-poly models are categorized by their level of refinement: high-poly models emphasize detail and fine structure, while low-poly models omit details and emphasize the overall structure.

[0184] In indoor scenes, the transparency of the building's interior walls is controlled according to the character controller's position. When the character controller reaches a certain floor of the building, the walls of the house on that floor are set to translucent, allowing the character controller to intuitively see the internal structure of the building.

[0185] 3) Collision Detection Mechanism

[0186] Through mathematical calculations, we can determine whether two non-penetrating objects in the space intersect. The collision detection in this scene is converted into the intersection problem of the geometric model.

[0187] The bounding sphere is the smallest sphere that contains the object. By directly detecting whether the distance between the center of the circle is less than the radius, we can detect whether the objects intersect.

[0188] In 3D space, AABB is a box-shaped cuboid with the normal of each face parallel to a given coordinate axis. Only the relationship between the corresponding coordinate axes is compared to detect whether there is an intersection. OBB is the smallest cuboid that contains an object and is relative to any coordinate axis direction. It has directionality. If the projections of two OBBs on all projection axes (perpendicular lines to the OBBs themselves) overlap, it is determined to be a collision; otherwise, it is determined that no collision has occurred.

[0189] 3. 3D map building indoor and outdoor integrated visualization system

[0190] (1) Prototype system composition

[0191] 1) System Architecture

[0192] Developed based on Unity engine, published on web platform using browser / server mode B / S structure, the system architecture is as follows Figure 6 As shown, it consists of three parts, including presentation layer, business logic layer and data layer, with the core part in the presentation layer;

[0193] The data layer includes the raster map database and vector data database provided by the Mapbox map server; the model relational database that stores model properties and model file paths; and the model resource files of the 3D building model.

[0194] The business logic layer receives client requests, such as requests for sliced ​​data of a specified scale in a specified area, parses the request parameters, initiates a query to the database, and finally encapsulates the query results and returns them to the presentation layer.

[0195] The presentation layer responds to various user operations, parses and renders the 3D building model data, 3D map model data, and path data returned by the control layer, realizes visualization, and interactively operates the map: character roaming and path query.

[0196] II) Overall process of indoor and outdoor integrated visualization system

[0197] Visualization process such as Figure 7 shown.

[0198] Step 1: Get the map area based on the current visible area range;

[0199] Step 2: The web page sends a request to obtain vector slices and Ou Ge slice maps. The remote server requests the database to obtain raster images and binary files encoded by the vector slices, and returns the data to the web page. The web page decodes the received binary slice files and then renders the 3D map layer. While requesting the map data, the database is requested to query the 3D building models within the current map range and request the model resource files.

[0200] Step 3: Build a 3D map model and render it in layers;

[0201] Step 4: Load the 3D building LOD model in the visible area by layer;

[0202] Step 5: If the user's operation causes the visible area to change at this time, return to the first step to request and render again. If the visible area does not change, this process ends, realizing indoor and outdoor integrated 3D visualization.

[0203] (2) Module design

[0204] 1) Module composition

[0205] For indoor and outdoor integrated 3D visualization and 3D roaming, it is divided into indoor and outdoor integrated visualization module, roaming module and query module. Each module has several sub-modules, such as Figure 8 shown.

[0206] 2) Visualization Module

[0207] Retrieve raster slices, vector slices, and building models through HTTP requests, decode the vector slice data, enable user zoom and pan interactions, dynamically render maps of the visible area, and achieve integrated indoor and outdoor visualization layered rendering. Render raster data, vector data, and LOD models in layers using a layered approach.

[0208] 3) Roaming Module

[0209] Three sub-modules are set up: free roaming, character roaming and small map; the system increases user immersion, in addition to free roaming, there are first-person and third-person roaming.

[0210] Switching between free roaming and character roaming means switching the camera and controlling the corresponding camera SetActive property to false or true.

[0211] 1) Free roaming: Users can observe the 3D scene from any angle by operating the camera themselves. They can control the rotation of the camera by right clicking the mouse, and the left, right, and forward movement of the camera by using the WSAD keys on the keyboard, that is, the movement of the viewing angle. The latitude and longitude of the intersection of the window position ray (a ray that passes through the viewpoint from the camera) and the map plane are determined based on the current camera position. The map slice code facing the current camera can be obtained. According to the set slice visual range, the slice data of the corresponding slice number is requested. The requested slice number will be marked as active. After the map is operated and the camera is moved, the map slice range will change. At this time, if the requested slice is active, that is, the existing slice does not need to be requested again. If the value of the slice and the center slice exceeds the destruction range value, it does not need to be placed in memory and will be destroyed immediately.

[0212] 2) Character roaming: Figure 9 It is the process of creating a character walkthrough.

[0213] Step 1: Add a terrain collider to the ground;

[0214] Step 2: Add a first-person controller. First, create an empty GameObject called FPSController. Then, add a rigidbody control under this GameObject to give it gravity so that it is controlled by gravity. Next, add a character control script so that the user can control the object's movement, forward and backward, left and right, as well as walk and run, using the mouse and keyboard. Finally, add a new GameObject called Camera under the FPSController object to act as the user's eyes and follow the controller's movements.

[0215] Step 3: Add a third-person controller. To see the mission model from a third-person perspective, add a mission model to the controller, along with a rigid body and colliders to achieve a realistic physical world effect. Also, add a character control script and a camera follow script. As the character model moves under user control, the camera will continuously follow the character, ensuring it always stays behind them. Finally, place the main camera in the third-person controller and bind the camera to the character.

[0216] Step 4: Add a perspective switching button to the system interface. When switching perspectives, not only the character controller is switched, but also the map camera needs to be replaced, allowing users to observe the scene from different perspectives.

[0217] 3) Minimap: Displays the surrounding environment information of the roaming protagonist, with the roaming protagonist as the center. This is achieved by removing the main camera, adding an additional camera, and setting this camera to perspective projection. Perspective projection increases the projection area, and finally displays the content of the projection area on the UI control.

[0218] According to the process, first, add a new camera to the scene and name it to distinguish it from the main camera, so that the roaming character is always in the center of the small map. Set the camera as a child object of the roaming character and set its coordinates above the character model. Aim the camera at the character model. The picture captured by the camera is the picture in the small map. Then display the camera picture on the render texture of the created Canvas. At this time, the camera picture is displayed as a control on the interface.

[0219] To make the roaming character more conspicuous in the minimap, different layers need to be set. First, add a model such as a sphere to the roaming character. The main camera shoots the character model, while the character model is displayed as a sphere in the minimap. Add two layers, a minimap layer MinimapLayer and the other is the character model layer PersonLayer. Then set the layer of the sphere to the newly added layer MinimapLayer; set the roaming character model to the PersonLayer layer, and finally modify the corresponding camera rendering layer. The main camera renders the PersonLayer layer instead of the MinimapLayer layer, while the minimap camera does the opposite.

[0220] 4) Navigation module

[0221] The outdoor navigation function is implemented in the system, that is, the starting point is selected in the interactive interface and the navigation path is displayed on the 3D map. The implementation process is as follows:

[0222] Step 1: Design the query panel on the interface;

[0223] Step 2: Send an HTTP request, and the server returns the navigation path with data in JSON format;

[0224] Step 3: Create a new game object as the displayed path;

[0225] Step 4: Parse JSON and use the parsed geometry data as the geometric attributes of the object;

[0226] Step 5: Add the rendering style of the navigation path.

[0227] 4. Example Verification

[0228] The 3D building model of this application is a four-story building. The interior is composed of various types of indoor spaces, such as rooms, connections, and passages. According to the two-dimensional layout of the laboratory, the indoor 3D model is organized according to the division of the 3D indoor space structure to construct the following Figure 10 Then, a series of LOD models are formed according to the 3D building LOD model construction rules.

[0229] Indoor and outdoor integrated display:

[0230] According to the change of map scale, the building model changes from coarse to fine. During this change, the visualization effect of indoor and outdoor integration is achieved. Figure 11 As shown, (a) shows that when the percentage is greater than 60% and less than 90%, the building model is a LOD1 hierarchical model; (b) shows that when the percentage is greater than 40% and less than 60%, the building model is a LOD2 hierarchical model.

Claims

1. A method for visualizing indoor and outdoor buildings on a smart city 3D map, characterized in that: Expand the representation of 3D map models and building scene divisions, and establish a representation method for 3D map models and 3D building models suitable for integrated indoor and outdoor 3D visualization: the 3D map model uses the DLG + grid base map + DEM representation method, and overlays the building model of FBX data through the coordinate connection relationship between the building and the map; at the same time, construct a 3D building LOD model, the map data comes from the Mapbox map server, and adopts a key file system + relational database to store the 3D building model; Set up the data transmission process, including the transmission process from the server to the client and the process from the client requesting data to rendering; Establish an integrated indoor and outdoor rendering method: perform layered rendering based on Mapbox, and dynamically render according to the visible area; implement a roaming mechanism after visualization, so that users can switch from outdoor scenes to indoor scenes through roaming; and realize the integrated indoor and outdoor visualization application of urban 3D map buildings; 1) Indoor and outdoor integrated 3D model organization and expression: Construct 3D map representation content and 3D map model representation methods, taking into account the representation of the base map and the vectorized representation of the ground objects, and satisfying the representation of later interactive operations; construct the outdoor and indoor representations of the 3D building model, build the 3D building model based on the representation method, and then associate the 3D map model and building model through coordinate data based on the topological relationship between the two; construct a multi-level of detail model representation and design the construction rules of the 3D building LOD model; 3D data organization method based on Mapbox web map service: for vector data, the vector slice data MVT structure is used to store model resource files on the server, and a relational database is used to store 3D building model information, which can be retrieved based on key-value queries; HTTP data access based on Unity: Using desktop browser as 3D visualization client, WebGL as graphics rendering engine API, and Unity's communication class for network data transmission for web page network communication; Dynamic rendering and caching design are adopted on the client to achieve rapid data acquisition and reduce the pressure of real-time rendering; 2) Integrated indoor and outdoor 3D visualization: Indoor and outdoor layered rendering: Indoor and outdoor data are rendered in layers based on the Mapbox toolkit, and dynamic rendering is performed based on the visible area; Interactive roaming: Based on Unity's roaming mechanism, indoor and outdoor scenes are switched through user roaming. The roaming mode adopts free roaming and interactive roaming. Users control the roaming position and direction according to their intentions.

2. The method for visualizing indoor and outdoor buildings on a smart city 3D map according to claim 1, characterized in that: The 3D model of indoor and outdoor integration: 1) 3D map model representation: A 3D map model is created using a combination of a digital line map (DLG) + a grid base map + a digital elevation model (DEM). This allows for integrated 3D visualization of both indoor and outdoor areas, with detailed representation of land features, especially buildings. The outdoor environment and landforms are also reflected to enable roaming. 2) 3D building model representation: The building's exterior outline and interior objects together constitute the building as a whole. The 3D building model only represents the building's geometric structure and surface texture. The 3D building model representation is divided into exterior representation and interior representation. The integrated interior and exterior 3D building model includes both the building's exterior outline and interior objects. 1) Outdoor Representation: 3D building models are built using geometric models. The outdoor representation is the building's outer contour, which consists of two parts: geometric structure and surface texture data. The model's geometric structure includes the building's main body, roof, and ancillary facilities. Based on the geometric structure, texture images are added to the building model. 2) Indoor representation: including rooms, passages, connections, and indoor entities; 3) Integrated indoor and outdoor representation: The outdoor 3D map model and 3D building model are displayed simultaneously. The 3D building model containing the building's outer contours and indoor scenes is placed in the outdoor 3D map model according to its actual geographic coordinates in reality. Through the integrated indoor and outdoor representation, both the indoor 3D scene and the outdoor surrounding environment scene can be observed.

3. The method for visualizing indoor and outdoor buildings on a smart city 3D map according to claim 1, characterized in that: Integrated indoor and outdoor data organization: 1) Ground Data Organization: 3D map models are implemented using a combination of a digital line map (DLG) + a raster basemap + a digital elevation model (DEM). The DEM data comes from raster data, and the raster basemap uses raster slice data. When the client requests raster slices, the server renders them using Mapnik. The DLG representing ground features uses vector slice data, sourced from MVT data in the Mapbox map service. The terrain raster data format is Mapbox's Terrain-RGB. The data contains global elevation data encoded in raster slices. The RGB color values ​​in the slices are decoded into the original height in meters. A DEM is constructed based on the elevation data. The DEM data is organized and represented in a regular rectangular grid. Regular points are collected on the slices to form a 3D terrain model of the regular rectangular grid. The surface raster slice data is then overlaid on the 3D terrain model. The DLG representing the ground features uses the street vector slice data MVT. The vector slices are organized by layers. Each layer is named according to the ground feature it corresponds to. It contains two parts: geometric features and metadata information. The slices contain layers, and each layer stores the corresponding ground features. Each vector tile stores at least one layer, and each layer stores at least one feature. Each feature encodes geometry and feature attributes in different ways. Vector tiles convert geographic coordinates into vector tile grid coordinates. The origin of the coordinate system is located in the upper left corner of the tile, with the X axis pointing rightward and the Y axis pointing downward. Points, lines, and polygons are encoded as x / y pairs. The integer pairs in the tag field correspond to feature attributes. In each tag pair, there are key and value pairs, respectively. The first integer is the index number of the key in the keys list of the layer to which it belongs. Similar to the first integer, the second integer is the index number of the value in the values ​​list. The index of each feature key is unique, ensuring that each feature does not have duplicate attributes. 2) Data Organization of 3D Building Models 3D models use the FBX format and a SceneGraph / Tree structure to store all model information. The root node of the file scene tree contains geometric networks, cameras, light sources, and skeleton nodes, and the corresponding attribute information is read by traversing; The integrated indoor and outdoor 3D building model is divided into the outer contour model and the indoor object model. The 3D building model is constructed with LOD. The representation of a building will have a set of 3D building models of different precisions, each of which contains one or more FBX files. The building is divided into the building outline and interior space, where the building outline includes the roof and exterior walls; the interior space is organized according to the divided elements; the 3D model data of the interior space is organized according to the topological relationship of the interior space.

4. The method for visualizing indoor and outdoor buildings on a smart city 3D map according to claim 1, characterized in that: Server-to-client data transmission process: Map data is pre-generated into a data pyramid and stored in blocks on the server. Users send network requests with parameters to the server to obtain map slices of a specified scale and location, where the URL and resource correspond one to one; The workflow for requesting Mapbox's tile map service is as follows: (1) The client sends a request: The client sends an HTTP request. The requested tile URL format is http: / / ip:port / version / {map_id} / {z} / {x} / {y}.filenameSuffix?param=value; where version is the version of the current map data; filenameSuffix is ​​the file suffix, i.e. the format of the request data; {z}{x}{y} are tile map parameters, {z} is the scale of the tile map, and {x}{y} is the index value of the tile. The tile map parameters are calculated based on the geographic coordinate value and the display scale. (2) The remote server parses the path to obtain the coordinates and extension suffix: if there is a cache, read the slice in the cache; if there is no data in the cache, obtain the specified map slice according to the URL format; (3) Execute the query: Calculate the range based on the coordinates and generate slices from the query data; a) First connect to the database; b) Execute the query statement and write the query results into the newly created vector tile; If the request is for raster tiles, render the vector tiles into raster tiles through the API; c) close the database; d) save the newly created tiles and save the tiles to the cache database; e) Send the data to the client.

5. The method for visualizing indoor and outdoor buildings on a smart city 3D map according to claim 1, characterized in that: The client-side process from request to rendering is the process from client interaction with the map to map rendering updates. If the intermediate data source changes or the visible tile set changes, the changed tiles need to be obtained and decoded and then passed into the buffer for rendering. (1) Event triggering and tile updating: The client process from request to rendering starts when the user initiates a request event. When the map is initialized, Clear is first used to clear all currently activated tiles, and then OnInitialized and OnTileAdded of the tiles are triggered to add the tiles at the current location. When the map is updated, OnTileAdded and OnTileRemoved are triggered to add tiles within the current visible area and remove tiles outside the visible area. When the map is zoomed or moved, the tile set within the visible area is recalculated, and the updateMap() method of the Map class is called to update the map. (2) Requesting tile resources: The tile set maintains the tiles within the current visible range. After the update is successful, the raster and vector resources on the tile are requested through the IFileSource interface according to the tile CanonicalTileld. The request is an HTTP request. After requesting the server, the resources are returned and the layer is initialized. When sending an HTTP request in Unity to obtain data on the map server, the UnityWebRequest library class is used. The request method consists of three elements. (3) Tile decoding: The data transmitted from the server is serialized binary format data encoded in the vector tile transmission format. The tile data stored in the local cache is also intact binary format data. The original data is decoded and used to build the model. VectorTile calls VectorTileReader to read the tile data for decoding, i.e. serialization, and processes the stored attribute information and coordinate information. A tile contains multiple geographic feature layers, and the layers are read and decoded in a loop, and the features in the layers are decoded one by one. Then the decoded data is visualized in layers.

6. The method for visualizing indoor and outdoor buildings on a smart city 3D map according to claim 1, characterized in that: Mapbox-based layered rendering: Layers are visualized in layers, divided into surface layers and building layers. The surface layer is represented by a 3D terrain model consisting of a terrain network and surface textures. The feature layer is divided into an indoor and outdoor integrated building layer represented by a 3D building model, and a road layer, general building layer, and land cover layer visualized by MVT data. Use the following scripting interface for layer visualization: (1) MapVisualizer: implements map visualization, creates requested map slices and relays them to the required Factory, reuses slices based on a caching mechanism, and locates slices in the Unity world; (2) Factory: Processes the map's terrain, images, and vector data for rendering; (3) LayerVisualizer: receives vector data and creates and styles features (such as buildings) as layers on the map; (4) Stacks and Modifiers: Stacks are a collection of Modifiers. Modifiers are scripts that create, modify, and style features. This implementation of MapVisualizer requires setting up several Factories: (1) TerrainFactory, requests and creates the world base grid, i.e., the terrain grid; (2) MapImageFactory, requests map raster images and provides materials and textures to TerrainFactory; (3) VectorTileFactory, which sets up a relatively complex interface in the Factory that processes vector data; In VectorTileFactory, you can request one or more of the following vector data types: streets, terrain, and traffic. The request will merge the datasets and return a data package. Each data source is visualized as a layer. The traffic dataset layer uses lines of different colors to represent traffic flow conditions. The terrain dataset layer has three source layers: a terrain overlay layer, a hillside layer, and a highline layer. The street dataset has the largest amount of data and consists of various urban elements. Multiple data sources are visualized using one layer. To maintain the orderliness and intuitiveness of the data organization, each data source is visualized on its own layer. The layer visualizer distributes the received vector data to the modifier stack for processing by layer. The modifier stack contains two types of modifiers, game object modifiers and mesh modifiers, which are used to create and decorate game objects. The mesh modifier generates the data required for the game object mesh. First, all mesh modifiers are run to generate all the data required for the mesh, and the mesh data is used to create the game object. Then all game object modifiers are run to decorate the game object.

7. The method for visualizing indoor and outdoor buildings on a smart city 3D map according to claim 1, characterized in that: Indoor and outdoor integrated rendering process: After the data is processed by the CPU and GPU from the memory, the image is displayed using the video memory; Rendering of 3D scenes is based on a programmable pipeline. The process is to draw 3D objects on a 2D screen. The visualization result is to convert the mathematical vertices into pictures, and the user sees a series of 2D images. Convert 3D objects into 2D images and transfer data from the CPU to the GPU. The data transferred includes 3D object information, viewpoint information, light source information, lighting model, and texture data. Finally, a 2D image is rendered. The rendering pipeline is divided into three stages. 1) Application stage: Data is transferred from memory to the CPU for calculation. The memory and CPU transmit the data generated after a series of calculations in Unity, such as frustum clipping, scene graph creation, and collision detection, to the GPU through the data bus for processing in the next stage. The generated vertex coordinates and texture coordinates are used in the next vertex processing stage, and the texture information is used in the rasterization stage. 2) Geometry phase: This is performed on the GPU and is responsible for 3D vertex coordinate transformation and lighting calculation. During this phase, vertex coordinate transformation, screen clipping, lighting attribute processing, projection transformation, and screen mapping are performed in sequence. Finally, the transformed and projected vertex coordinates, texture coordinates, and colors are calculated. Get the screen coordinates of the object's vertices, and then use the original connection relationship between the vertices to calculate the mesh structure of the object. The index and vertex are the two major components of the mesh. The latter is linked to the former to form the basic geometric unit, which is then clipped to retain only the content within the screen range. The converted vertex coordinates and fragments required for the next stage of drawing are output; 3) Rasterization stage: The color and depth values ​​corresponding to each pixel in the fragment are calculated using the texture information generated by the application stage. The color value of the pixel will be written and saved in the frame buffer; The steps of the rasterization stage are: rasterizing and interpolating the fragment and vertex coordinates, and then performing pixel operations and pixel frame buffering; the 3D model is projected onto a two-dimensional plane through vertex transformation and assembled into primitives.

8. The method for visualizing indoor and outdoor buildings on a smart city 3D map according to claim 1, characterized in that: Indoor and outdoor roaming: 1) Interactive roaming: Users use character controllers to interactively roam in a 3D virtual environment, observing the virtual scene from the character controller's perspective. During the roaming process, they analyze the scene and calculate the path. By continuously moving the viewpoint or changing the direction of sight, and using the mouse and keyboard as system input devices, users can interact with the virtual scene. 2) Indoor and outdoor scene switching: On the one hand, users can operate through free roaming to observe indoor and outdoor scenes from any angle; or use character roaming to control the character controller to walk outdoors to observe the surrounding environment and enter indoor scenes for observation. On the other hand, when switching from outdoor to indoor, the level of detail of indoor and outdoor building objects is achieved by controlling the resolution. The level of detail of the building model changes from coarse to precise, and the visibility of the building objects changes from non-existent to present. Indoor and outdoor objects change from visible to invisible according to the percentage of the building model in the view. In indoor scenes, the transparency of the building's interior walls is controlled based on the character controller's position. When the character controller reaches a certain floor of the building, the walls of that floor are set to semi-transparent, allowing the character controller to intuitively see the building's interior structure. 3) Collision detection mechanism: This mechanism uses mathematical calculations to determine whether two non-penetrating objects in a spatial range intersect. Collision detection in this scenario is converted into an intersection problem of geometric models. The bounding sphere is the smallest sphere that contains the object. By directly detecting whether the distance between the center of the circle is less than the radius, we can detect whether the objects intersect. In 3D space, AABB is a box-shaped cuboid with the normal of each face parallel to a given coordinate axis. Only the relationship between the corresponding coordinate axes is compared to detect whether there is an intersection. OBB is the smallest cuboid that contains an object and is relative to any coordinate axis direction. It has directionality. If the projections of two OBBs on all projection axes overlap, it is determined to be a collision; otherwise, it is determined that no collision has occurred.

9. The method for visualizing indoor and outdoor buildings on a smart city 3D map according to claim 1, characterized in that: 3D map building indoor and outdoor integrated visualization system composition: 1) System Architecture: Developed based on the Unity engine and published on a web platform, it adopts a browser / server model B / S structure and consists of three parts: the presentation layer, the business logic layer, and the data layer, with the core part being the presentation layer. The data layer includes the raster map database and vector data database provided by the Mapbox map server; Model relational database, storing model attributes and model file paths; And the model resource files of 3D building models; The business logic layer receives client requests, such as requests for sliced ​​data of a specified scale in a specified area, parses the request parameters, initiates a query to the database, and finally encapsulates the query results and returns them to the presentation layer. The presentation layer responds to various user operations, parses and renders the 3D building model data, 3D map model data, and path data returned by the control layer, and realizes visualization and interactive map operations: character roaming and path query; II) Overall process of indoor and outdoor integrated visualization system Step 1: Get the map area based on the current visible area range; Step 2: The web page sends a request to obtain vector tiles and Ouge tile maps. The remote server requests the database to obtain raster images and binary files encoded by the vector tiles, and returns the data to the web page. The web page decodes the received binary tile files and then renders the 3D map layer. While requesting map data, request the database to query the 3D building model within the current map range and request the model resource file; Step 3: Build a 3D map model and render it in layers; Step 4: Load the 3D building LOD model in the visible area by layer; Step 5: If the user's operation causes the visible area to change at this time, return to the first step to request and render again. If the visible area does not change, this process ends, realizing indoor and outdoor integrated 3D visualization.

10. The method for visualizing indoor and outdoor buildings on a smart city 3D map according to claim 1, characterized in that: Roaming module: It has three sub-modules: free roaming, character roaming and mini-map. It increases user immersion in the system. In addition to character-free roaming, it also provides first-person and third-person roaming. Switching between free roaming and character roaming means switching the camera, and controlling the corresponding camera SetActive property to false or true; 1) Free roaming: Users can observe the 3D scene from any angle by operating the camera themselves. They can control the camera's rotation by right-clicking the mouse, and the camera's forward, backward, left, and right movements by pressing the WSAD keys on the keyboard, i.e., the angles of the viewing angle. Based on the current camera position, the latitude and longitude of the intersection of the viewport ray (a ray that passes through the viewpoint) and the map plane are determined. This determines the map slice code facing the current camera, and based on the set slice visibility range, requests slice data for the corresponding slice number. The requested slice number is marked as active, and the map slice range changes after map operations and camera movements. At this time, if the slice to be requested is active, that is, the existing slice does not need to be requested again; if the value of the slice and the center slice exceeds the destruction range value, it does not need to be placed in the memory and occupies memory, and is destroyed immediately; 2) Character roaming: Step 1: Add a terrain collider to the ground; Step 2: Add a first-person controller. First, create an empty GameObject called FPSController. Then, add a rigidbody control under this GameObject to give it gravity so that it is controlled by gravity. Next, add a character control script so that the user can control the object's movement, forward and backward, left and right, as well as walk and run, using the mouse and keyboard. Finally, add a new GameObject called Camera under the FPSController object to act as the user's eyes and follow the controller's movements. Step 3: Add a third-person controller to see the mission model from a third-person perspective. Add a mission model under the controller, add a rigid body and a collider to achieve a real physical world effect. At the same time, add a character control script and a camera follow script. When the character model moves under user control, the camera will continue to follow the character, ensuring that the camera is always behind the character to shoot. Finally, place the main camera in the third-person controller and bind the camera and character together. Step 4: Add a perspective switching button to the system interface. When switching perspectives, not only the character controller is switched, but also the map camera needs to be replaced, allowing users to observe the scene from different perspectives. 3) Minimap: Displays the surrounding environment information of the roaming protagonist, with the roaming protagonist as the center; The implementation method is to remove the main camera, add an additional camera, and set this camera to perspective projection. Perspective projection increases the projection area, and finally display the content of the projection area on the UI control; First, add a new camera to the scene and name it to distinguish it from the main camera. Make sure the roaming character is always in the center of the mini-map. Set the camera as a child object of the roaming character and set its coordinates above the character model. Aim the camera at the character model. The image captured by the camera will be the image in the mini-map. Then display the camera image on the render texture of the created Canvas. At this point, the camera image is displayed as a control on the interface. To make the roaming character more conspicuous in the minimap, different layers need to be set. First, add a model such as a sphere to the roaming character. The main camera shoots the character model, while the character model is displayed as a sphere in the minimap. Add two layers, a minimap layer MinimapLayer and the other is the character model layer PersonLayer. Then set the layer of the sphere to the newly added layer MinimapLayer; set the roaming character model to the PersonLayer layer, and finally modify the corresponding camera rendering layer. The main camera renders the PersonLayer layer instead of the MinimapLayer layer, while the minimap camera does the opposite.

Images