A method and system for reducing real historical style reproduction of urban planning display

Abstract

The application discloses a kind of for city planning display's loss reality historical style reappearance method and system.The system is in accordance with the display needs of city planning, through the collaborative work of five modules such as semantic perception and anchor extraction, multi-source space-time positioning, loss reality repair, generative style reappearance and virtual-real fusion interaction, realize the accurate in-situ reappearance of city historical style.The application breaks through the limitations of static graphics, sand table in traditional city planning display, erases modern interference information through loss reality technology, solves the visual conflict of ancient and modern interlaced and the positioning drift problem under the large-scale scene of outdoor, can be widely applied to city planning exhibition, historical and cultural block protection and immersive travel experience, provides the time travel experience of what you see is what you get for public, helps visitor to establish complete city evolution space-time view.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, augmented reality (AR), and urban planning display technology. Specifically, this invention relates to a method and system for reproducing the historical appearance of a city, specifically designed for urban planning display scenarios. This method utilizes deep learning for semantic analysis, removes modern interference through diminished reality (DR), and combines generative AI to reproduce historical scenes. Background Technology

[0002] A city is not merely a collection of physical spaces, but also an overlay of slices of time. As a crucial window showcasing the trajectory of urban development, the core mission of urban planning exhibition halls is to help the public understand "where the city came from and where it is going." However, existing methods of showcasing urban planning often suffer from the following technical bottlenecks: 1. Historical temporal and spatial fragmentation: Existing exhibitions mostly rely on static images and texts or sand tables, lacking a continuous display based on a timeline, making it difficult for visitors to form a complete spatiotemporal understanding of urban evolution. In particular, static displays cannot intuitively present the historical changes of "sea change" in terms of changes in urban spatial form.

[0003] 2. Conflict between virtual and real objects: Traditional outdoor augmented reality technologies often use a simple overlay method, directly rendering virtual historical buildings onto the camera's view. In complex city streets, modern objects (such as cars, pedestrians, and modern high-rise buildings) often create visual conflicts with the virtual historical scene, resulting in clipping and severely damaging the immersive experience and display effect.

[0004] 3. Insufficient positioning accuracy: In urban canyon environments, relying solely on GPS cannot meet the requirements of centimeter-level AR overlay. Furthermore, traditional visual SLAM technology is prone to losing tracking in outdoor environments with large changes in lighting and repetitive textures, causing virtual scenes to float or shake, affecting the accuracy and professionalism of the display.

[0005] 4. Lack of intuitive comparison between past and present: Current technology makes it difficult to intuitively present the in-situ comparison between "historical planning" and "the current situation in various modern periods" as well as "future planning schemes." Visitors in modern urban spaces find it difficult to intuitively perceive the evolution of the geographical attributes of their current coordinates in different historical periods (such as changing from a river to a road), and also cannot intuitively understand the impact of future planning on the urban spatial form.

[0006] 5. Poor interactivity: Existing exhibits are mostly one-way lectures, lacking interactivity and a sense of participation. Especially in scenarios with multiple visitors, guides cannot uniformly control the content and viewing angles, resulting in low exhibit efficiency.

[0007] In summary, there is an urgent need for a dedicated urban planning display method that can integrate semantic understanding, accurately handle virtual and real occlusion, and realistically restore historical features. This invention introduces semantic segmentation and reduced reality technology to visually "erase" modern interference, achieving a true "in-situ historical journey," and designs dedicated interactive functions for the specific needs of urban planning display. Summary of the Invention

[0008] This invention provides a system for recreating the diminished historical appearance of a city through urban planning displays, comprising: The semantic perception and anchor point extraction module includes an image preprocessing subunit, a semantic segmentation inference subunit, and an anchor point selection subunit. The semantic segmentation inference subunit is equipped with an improved DeepLabV3+ deep neural network, using MobileNetV2 as the backbone network and adding a channel attention mechanism to identify categories such as sky, buildings (modern / historical), roads, vegetation, pedestrians, vehicles, and billboards at the pixel level from real-time video streams. The anchor point selection subunit extracts natural skylines and historical relics as "anchor points" for visual localization based on semantic labels, and identifies modern vehicles, pedestrians, and modern buildings as "features to be reduced."

[0009] Multi-source spatiotemporal positioning module: Connected to the semantic perception and anchor point extraction module, it includes a GNSS / IMU fusion subunit, a skyline matching subunit, and a visual inertial odometry subunit. The skyline matching subunit corrects GPS positioning errors and magnetometer azimuth errors by calculating the chamfer distance between the real skyline extracted in real time and the virtual skyline generated from GIS data. The visual inertial odometry subunit is used for pose tracking in urban canyon areas with indistinct skyline features and calibrates roll angle and altitude direction errors.

[0010] The Loss-Induced Reality Restoration Module, connected to the Semantic Awareness and Anchor Point Extraction Module, includes a dynamic object removal subunit and a static background completion subunit. The dynamic object removal subunit utilizes optical flow information from the past 10 frames to repair holes caused by moving objects. The static background completion subunit employs a two-stage structure-aware generative adversarial network. The first stage restores the geometric structure of the missing areas, and the second stage generates texture details. While preserving anchor point features, it intelligently fills in modern feature regions to generate a background texture that conforms to perspective.

[0011] The generative landscape reconstruction module includes a historical semantic matching subunit, a style transfer generation subunit, and a lighting estimation subunit. The historical semantic matching subunit supports multi-time-node data queries from the Ming and Qing dynasties to the modern era. The style transfer generation subunit utilizes a SPADE conditional generative adversarial network to convert the restored scene and historical semantic layout map into architectural styles specific to a particular historical period. The lighting estimation subunit estimates ambient lighting in real time based on a second-order spherical harmonic function, regressing 9 coefficients per color channel, for a total of 27 coefficients, ensuring consistency between the lighting and shadows of virtual historical objects and the real environment.

[0012] The virtual-real fusion interactive module includes a rendering engine subunit, a user interface subunit, and a multi-user collaboration subunit. The rendering engine subunit supports multi-level alpha blending. The user interface subunit provides four interactive methods for urban planning presentations: a timeline slider, key event node navigation, a side-by-side comparison of past and present data, and overlay of planning schemes. The multi-user collaboration subunit supports simultaneous display on the guide's terminal and multiple viewer terminals.

[0013] This invention also discloses a method for reproducing the historical appearance of a city in a way that minimizes damage, based on the aforementioned system, comprising the following steps: S1: Multi-scale semantic perception and anchor point extraction: Acquire real-time images from mobile terminals and generate pixel-level classification masks using an improved DeepLabV3+ model. Based on preset rules, invariant elements such as the sky, mountains, and historical buildings are extracted as positioning anchor points, while elements such as modern vehicles, asphalt roads, and modern buildings are marked as areas to be removed.

[0014] S2: Visual-Geographic Joint Positioning and Attitude Calibration: This function integrates coarse GPS coordinates and IMU attitude data to retrieve the DEM elevation model and city white model from the current viewpoint. By minimizing the chamfer distance between the real-time skyline and the DEM projected skyline, the precise yaw and pitch angles of the camera are calculated. Combined with visual-inertial odometry calibration of roll angle and altitude orientation error, accurate estimation of the six-degree-of-freedom pose is achieved.

[0015] S3: Lossy Reality Mask Generation and Structure-Aware Inpainting: A binary occlusion mask is generated based on the semantic segmentation results. Dynamic occlusions are repaired using video spatiotemporal consistency, and static occlusions are repaired using a two-stage structure-aware inpainting algorithm, outputting an empty stage image free from modern interference.

[0016] S4: Historical Landscape Generation and Illumination Consistency Rendering: Based on the user-selected historical time point, historical GIS data and a historical building texture library are retrieved. A SPADE conditional generative adversarial network is used to transform the "empty stage" and historical semantic layout map into historical landscape images. The ambient lighting spherical harmonic coefficient is estimated, and shadows and highlights consistent with the current sunlight direction are added to the generated historical buildings.

[0017] S5: Virtual-Real Fusion Output and Interaction for Planning Display: This feature integrates generated historical images with preserved real-world anchor points for rendering. It supports multi-user simultaneous display controlled by guides, side-by-side display of ancient and modern comparisons, and overlay display of three-tiered planning schemes: "past-present-future".

[0018] The beneficial effects of this invention are as follows: Specifically designed for urban planning display scenarios: This invention not only solves the technical problems of general AR technology, but also addresses the specific needs of urban planning displays by designing dedicated functions such as multi-time-node historical data query, planning scheme overlay, and multi-user collaborative display. It can be directly applied to display activities in urban planning exhibition halls and historical and cultural blocks. By introducing subtractive reality technology, it achieves subtraction processing of modern interference information, fundamentally solving the visual pain point of "ancient and modern interweaving," making historical scenes more realistic and credible, and greatly enhancing the immersiveness and persuasiveness of urban planning displays. Through semantic-level skyline matching, it utilizes macroscopically unchanging contour features in the city for positioning. Compared with traditional texture feature point matching, it has higher robustness and positioning accuracy in environments with large changes in outdoor lighting and complex textures. Combined with visual inertial odometry, it solves the positioning problem in areas with indistinct skyline features. Background restoration is performed through a two-stage structure-aware GAN, ensuring seamless integration between the restored area and the surrounding environment. By utilizing SPADE conditional generative adversarial networks to generate historical landscapes, historical images with photorealistic quality can be produced at a lower computational cost compared to traditional 3D modeling and rendering. It offers various interactive features such as timeline sliders, key event node jumps, side-by-side comparisons of ancient and modern times, and overlay of planning schemes, and supports multi-user collaborative display, enabling the public to intuitively understand the historical evolution and future development direction of the city. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the architecture of the system for reproducing the reduced historical features of urban planning displays according to the present invention; Figure 2 This is a flowchart of the method for reproducing the real historical features in urban planning displays according to the present invention; Specific implementation methods

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described in the following description.

[0021] like Figure 1 As shown, a system for reproducing the historical appearance of reduced reality for urban planning display includes a semantic perception and anchor point extraction module, a multi-source spatiotemporal positioning module, a reduced reality restoration module, a generative appearance reproduction module, and a virtual-real fusion interaction module.

[0022] Semantic perception and anchor point extraction module: Runs on the NPU of a mobile terminal (such as iPad Pro or high-performance AR glasses). Employs a DeepLabV3+ model with MobileNetV2 as its backbone. Input resolution is adjusted to 512x512 to ensure an inference speed of 30FPS. Output semantic categories include: sky, vegetation, buildings (modern / historical), roads, pedestrians, vehicles, etc. The system has a pre-built anchor point library, prioritizing natural terrain (mountain horizon) and buildings marked as "historical heritage" as references for visual localization. Runs on the NPU of a mobile terminal (such as iPad Pro or high-performance AR glasses). Employs an improved DeepLabV3+ model with MobileNetV2 as its backbone, adding a channel attention mechanism to the hollow space pyramid pooling module to enhance the extraction capability of elongated edge features. Input resolution is adjusted to 512x512 to ensure an inference speed of 30FPS. Output semantic categories include: sky, vegetation, buildings (modern / historical), roads, pedestrians, vehicles, billboards, temporary structures, etc. The system has a pre-set anchor point library, prioritizing natural landforms (mountain horizons) and buildings and ancient trees marked as historical heritage as references for visual positioning.

[0023] Multi-source spatiotemporal positioning module: For coarse positioning, latitude and longitude are obtained using GNSS, and orientation is obtained using an electronic compass. For fine positioning, a skyline matching algorithm is used. The system loads the digital elevation model and city LOD1 white model of the surrounding area from an offline GIS database to render the theoretical skyline. The system calculates the real-time extracted semantic skyline pixel set. With the theoretical skyline pixel set chamfer distance between : ; By fine-tuning the camera's yaw and pitch angles using gradient descent, $D_{chamfer}$ is minimized, thereby achieving pixel-level geographic alignment.

[0024] The loss reduction and realism restoration module: For dynamic processing, for masks of pedestrians and vehicles, optical flow information from the past 5 frames is used to find unoccluded background pixels for filling. For static processing, a gated convolutional (GAN) network is used for modern buildings or signs. This network includes a coarse restoration generator and a fine restoration generator. The loss function combines reconstruction loss, adversarial loss, and style loss to ensure seamless integration of the restored area with the surrounding real-world environment, such as sky color and ground texture.

[0025] Generative landscape reconstruction module: Utilizing a conditional GAN ​​with a SPADE (Spatially-Adaptive Normalization) architecture. Inputs are the restored background image and historical semantic layout map. For illumination estimation, a lightweight CNN is used to regress the first three spherical harmonic coefficients from the current camera's video frames. When rendering historical building models, these coefficients are used to calculate the diffuse reflection component, ensuring that the illuminated surfaces of the virtual buildings match the real environment.

[0026] like Figure 2 As shown, the method for recreating the diminished historical appearance in urban planning displays includes the following steps: Step S1: Multi-scale semantic perception and anchor point extraction The semantic awareness module downsamples and normalizes each frame of the video stream before inputting it into the improved DeepLabV3+ model. The model outputs a class probability map, which is then used to obtain the final semantic mask via Argmax. Morphological operations (erosion and dilation) are used to optimize the mask edges. The recognition logic determines the occlusion as follows: if a pixel belongs to the sky and the pixels below it are not in the sky, it is marked as a skyline edge point; if a pixel belongs to a modern vehicle or pedestrian, it is marked as dynamic occlusion; if a pixel belongs to a modern building, billboard, or temporary structure, it is marked as static occlusion.

[0027] Step S2: Visual-Geographic Joint Localization and Attitude Calibration Upon system startup, the initial position and attitude are first acquired through GNSS / IMU fusion. If the GPS error exceeds 10 meters, the system prompts the user to scan for surrounding tall buildings or mountains. The positioning module uses the IMU's rotation vector to align the virtual camera in the approximate direction and render a virtual skyline. A coarse-to-fine search strategy is employed for skyline matching: first, coarse matching is performed within a 360-degree range in 10-degree increments to determine the azimuth angle interval; then, fine matching is performed within a ±5-degree range in 0.1-degree increments. When the chamfer distance is less than 5 pixels, the current pose is locked, and visual inertial odometry is used for frame-by-frame tracking.

[0028] Step S3: Generation of reduced-value reality mask and structure-aware repair Based on the semantic results of S1, a binary occlusion mask is generated (1 for removal, 0 for retention). For dynamically occluded regions, spatiotemporal filling is performed using optical flow information from the past 10 frames. For statically occluded regions, the original image and the mask are input into a two-stage structure-aware GAN network. The network first infers the edge structure of the missing region, then uses a contextual attention mechanism to generate texture details, outputting a repaired "empty stage" image.

[0029] Step S4: Historical Landscape Generation and Lighting Consistency Rendering The system queries a historical GIS database based on the current geographic coordinates and the user-selected historical time point to obtain vector planning data and historical building textures for that time point. This vector data is then rasterized into a historical semantic map. The restored background image and the historical semantic map are input into a SPADE conditional generative adversarial network to generate an image with historical texture. Simultaneously, the lighting estimation subunit calculates the spherical harmonics of the current environment and adjusts the lighting parameters of virtual objects to match the real environment.

[0030] Step S5: Virtual-Real Fusion Output and Interaction for Planning Display The generated historical images are alpha-blended with preserved anchor points (such as preserved ancient trees and distant mountains) in the original video stream. Specific alpha channel values ​​are set for semi-transparent objects (such as river water). In multi-user display mode, the operator's commands are synchronized to all viewer terminals via a local area network. The system also supports a three-layer overlay display of "past-present-future," intuitively presenting the complete development trajectory of the city.

[0031] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. A system for reproducing the diminished historical appearance of a city in urban planning displays, characterized in that, include: The semantic perception and anchor point extraction module acquires real-time video streams from the mobile terminal's camera. It utilizes a lightweight deep learning network to perform pixel-level semantic segmentation of the current frame, identifying and extracting invariant features in the scene as visual positioning anchor points. Simultaneously, it identifies modern dynamic and static features as areas to be reduced. The multi-source spatiotemporal positioning module acquires coarse geographic coordinates and six-degree-of-freedom attitude data. It retrieves corresponding digital elevation models and city white models from historical geographic information databases. By performing chamfered distance matching between the real skyline in the real-time image and the virtual skyline generated by projecting the digital elevation model, it calibrates the camera's yaw and pitch angles. Finally, it combines visual inertial odometry to calibrate the roll and altitude. The system includes: a directional error reduction and real-world restoration module, which generates a binary occlusion mask based on the area to be reduced, and combines optical flow information with a generative adversarial network to visually erase and restore the background texture of the modern feature areas covered by the mask, generating a clean background image without modern interference; a generative landscape reconstruction module, which retrieves historical urban semantic layout data and a historical building texture library based on the positioning coordinates, uses a conditional generative adversarial network to generate building and environmental textures with specific historical styles on the restored background, and performs lighting and shadow rendering in conjunction with real-time ambient lighting estimation; and a virtual-real fusion interaction module, which performs alpha blending and overlay of the rendered historical landscape layer and the invariant feature layer in the real scene, and provides a dedicated interactive interface for urban planning display, supporting multi-user collaborative display, planning scheme comparison, and historical evolution demonstration.

2. The system according to claim 1, characterized in that, The semantic perception and anchor point extraction module includes an image preprocessing subunit, a semantic segmentation inference subunit, and an anchor point selection subunit. The semantic segmentation inference subunit is equipped with an improved DeepLabV3+ network model, using MobileNetV2 as the backbone network, and adding a channel attention mechanism in the hollow space pyramid pooling module. The anchor point selection subunit is equipped with classification logic, defining the sky and mountain outlines as primary anchor points, historical buildings and ancient trees as secondary anchor points, and modern buildings, vehicles, pedestrians, billboards, and temporary structures as features to be reduced.

3. The system according to claim 1, characterized in that, The multi-source spatiotemporal positioning module includes a GNSS / IMU fusion subunit, a skyline matching subunit, and a visual inertial odometry subunit; the GNSS / IMU fusion subunit is used to acquire initial geographic coordinates and attitude data; The skyline matching subunit employs a coarse-to-fine search strategy, calibrating the camera's yaw and pitch angles by minimizing the chamfer distance between the real-time skyline and the virtual skyline; the visual inertial odometry subunit is used for pose tracking in urban canyon areas where skyline features are not obvious, and calibrates roll angle and altitude direction errors.

4. The system according to claim 1, characterized in that, The loss reduction reality restoration module includes a dynamic object removal subunit and a static background completion subunit; the dynamic object removal subunit adopts an optical flow guided algorithm to process moving pedestrians and vehicles using the spatiotemporal information of the past 10 frames. The static background completion subunit is configured with a two-stage structure-aware generative adversarial network based on gated convolution. The first stage generates the edge structure of the missing region, and the second stage generates texture details to repair large background areas obscured by modern buildings.

5. The system according to claim 1, characterized in that, The generative landscape reconstruction module includes a historical semantic matching subunit, a style transfer generation subunit, and an illumination estimation subunit. The historical semantic matching subunit is used to retrieve urban semantic layout data corresponding to the historical period based on the calibrated pose, supporting multi-time node data queries from the Ming and Qing Dynasties to the present. The style transfer generation subunit uses a SPADE conditional generative adversarial network, taking the historical semantic layout map and the restored background image as input, to generate a landscape image with the architectural style of the corresponding historical period. The illumination estimation subunit uses a convolutional neural network to regress the second-order spherical harmonic function coefficients from the current video frame, with 9 coefficients for each color channel, for a total of 27 coefficients, to calculate the diffuse illumination distribution and main light source direction of the virtual historical scene.

6. The system according to claim 1, characterized in that, The virtual-real fusion interaction module includes a rendering engine subunit, a user interface subunit, and a multi-user collaboration subunit. The rendering engine subunit supports multi-level alpha blending, setting different blending coefficients for objects with different transparency. The user interface subunit provides four interactive methods for urban planning presentations: timeline slider, key historical event node jump, ancient and modern comparison split-screen display, and planning scheme overlay. The multi-user collaboration subunit supports synchronous display between the narrator's terminal and multiple audience terminals, allowing the narrator to control the perspective and time nodes of all terminals.

7. A method for reproducing the diminished historical appearance of a city in urban planning displays, applied to the system described in any one of claims 1-6, characterized in that, Includes the following steps: Step S1: Multi-scale semantic perception and anchor point extraction Through the semantic perception and anchor point extraction module, the improved DeepLabV3+ model is used to segment real-time video frames and output semantic masks; invariant features such as natural skylines, mountains, and historical buildings are selected as localization anchor points, and modern buildings, vehicles, pedestrians, and billboards are marked as features to be reduced. Step S2: Visual-Geographic Joint Localization and Attitude Calibration The initial position and attitude are obtained by using a multi-source spatiotemporal positioning module and GNSS / IMU fusion; the yaw and pitch angles of the camera are iteratively optimized by calculating the chamfer distance between the real-time skyline and the DEM projected skyline in the GIS database; and the roll angle and altitude direction error are calibrated by combining visual inertial odometry to eliminate cumulative drift. Step S3: Generation of reduced-value reality mask and structure-aware repair The loss reduction reality restoration module generates a binary occlusion mask based on the features to be reduced; it uses spatiotemporal optical flow to fill the holes caused by dynamic objects; and it uses a two-stage structure-aware GAN network to restore static large-area occlusions and restore the geometric structure and texture details of the scene. Step S4: Historical Landscape Generation and Lighting Consistency Rendering The generative landscape reconstruction module retrieves urban semantic layout data for the corresponding historical period based on the calibrated pose; the SPADE conditional generative adversarial network is used to transform the historical semantic layout into realistic historical landscape images; the current ambient illumination is estimated based on the second-order spherical harmonic function, and the generated historical images are relit. Step S5: Virtual-Real Fusion Output and Interaction for Planning Display Through the virtual-real fusion interaction module, historical images are alpha-blended with the preserved real-world anchor layer; it supports multi-user synchronous display controlled by the guide, screen-by-screen display of ancient and modern comparisons, and overlay display of planning schemes.

8. The method according to claim 7, characterized in that, In step S1, the training loss function of the semantic segmentation model adopts a combination of weighted cross-entropy loss and DiceLoss, in which pixels of the skyline and historical building categories are given a weight that is 4 times higher than that of the background category, in order to solve the segmentation accuracy problem of thin edge features; the weight value is determined by performing a grid search on a dataset containing 1000 urban street view images.

9. The method according to claim 7, characterized in that, In step S2, the calibration process for skyline matching adopts a coarse-to-fine search strategy: first, coarse matching is performed in 10-degree increments within a 360-degree range to determine the azimuth angle range; Then, fine matching is performed within a range of ±5 degrees in steps of 0.1 degrees; when the chamfer distance is less than 5 pixels, the matching is considered successful and the current pose is locked.

10. The method according to claim 7, characterized in that, In step S3, the loss function of the two-stage structured perceptual GAN ​​network includes a weighted sum of pixel reconstruction loss (L1), adversarial loss, and perceptual loss, with weights of 0.6, 0.2, and 0.2, respectively. The perceptual loss is calculated based on the conv4_3 layer feature map of the pre-trained VGG-19 network to ensure that the repaired texture is semantically consistent with the surrounding environment.

Images