A method for large-scene dynamic reconstruction and virtual-real collaborative rendering for smart tourism digital twin

Abstract

The application discloses a kind of big scene dynamic reconstruction and virtual-real collaborative rendering method for wisdom tourism digital twin, it is related to tourism scene three-dimensional reconstruction and virtual-real rendering technical field, the present application adopts the way of unmanned aerial vehicle oblique photography and ground laser radar to collect multi-source data, and the initial three-dimensional geometric model of scene with high-resolution texture is constructed by space-time registration, based on neural radiation field, construct scene continuous implicit representation model, synthesize high-fidelity scene image under any new perspective, separate processing to dynamic foreground and static background, respectively trajectory prediction and incremental point cloud update, adopt cloud edge collaborative rendering pipeline, through view point preloading, realize seamless synchronous output of cloud and terminal rendering content.The present application gives consideration to big scene modeling efficiency and key area texture precision, cloud edge collaborative architecture reduces terminal hardware requirement, virtual-real light consistency design improves tour immersion, can adapt to various tourism scene digital transformation needs.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction and virtual-real rendering technology for cultural and tourism scenes, and in particular to a method for dynamic reconstruction and virtual-real collaborative rendering of large scenes for smart cultural and tourism digital twins. Background Technology

[0002] The smart cultural tourism industry is currently entering a critical stage of digital upgrading, especially with the continuous increase in demand for immersive and interactive experiences. Digital twins, as a core technology for the digital reconstruction of scenes, are widely used in scenarios such as virtual tours of scenic spots, immersive exhibitions, and cultural interactive dissemination. Traditional digital reconstruction of cultural tourism scenes often adopts a single acquisition method. It either relies on drone oblique photography to quickly model large-scale scenes, but the texture details of key cultural relics and historical buildings are not well restored, making it difficult to meet the needs of close-up viewing; or it uses manual modeling to achieve high-precision replication of local scenes, which has a long modeling cycle and high cost, making it difficult to cover the needs of full-area reconstruction of large cultural tourism scenes. At the same time, existing virtual-real fusion rendering solutions mostly adopt a single deployment mode. Either all rendering tasks are completed in the cloud, which is affected by network transmission latency, and the screen is prone to lag and slow loading when tourists rotate their view; or all rendering tasks are executed on the terminal, which is limited by the performance of mobile terminals, resulting in insufficient rendering accuracy for large scenes and making it difficult to achieve high-fidelity virtual-real fusion effects.

[0003] Currently, mainstream 3D reconstruction technologies for cultural and tourism scenes fall into two categories. One is photogrammetry-based 3D reconstruction technology. This technology generates 3D point clouds by matching feature points from multi-view images, and then completes the 3D model construction through meshing and texture mapping. It has high acquisition efficiency and low cost, making it suitable for rapid modeling of large-scale cultural and tourism scenes. However, due to the influence of lighting conditions and dynamic element interference during shooting, the generated models are prone to texture distortion and excessive noise. Furthermore, it can only support rendering from preset viewpoints, failing to meet the needs of tourists who wish to freely adjust their viewing angles. The other category is pre-built model rendering technology based on game engines. This technology imports manually constructed 3D models into the game engine in advance, using the engine's real-time rendering capabilities to achieve a fusion display of virtual and real content. It offers strong interactivity and low rendering latency. However, model updates require manual completion and cannot synchronize dynamic content such as tourist movement, lighting changes, and exhibition adjustments within the physical scene in real time. Moreover, large scenes have slow loading speeds, high model resource consumption, and are difficult to adapt to the operating requirements of mobile terminals. Existing cloud-edge-device collaborative rendering solutions mostly adopt fixed task allocation strategies, which cannot dynamically adjust task allocation according to the load of edge nodes. During peak periods, edge node overload can easily lead to screen stuttering. In addition, scene preloading strategies mostly use fixed range loading, which can easily waste bandwidth resources and cannot cover all the needs of the visitor's perspective rotation.

[0004] Current digital transformation efforts are mostly concentrated in leading scenic areas. Many historical sites and residences in remote areas still rely primarily on traditional static displays. Limited by hardware and construction costs, these sites struggle to deploy high-performance rendering equipment, failing to provide visitors with a consistently high-quality immersive experience. Furthermore, some scenic areas use pre-recorded, fixed-view videos with highly homogenized content, failing to meet visitors' personalized needs. The low matching rate between virtual content and real-world lighting and shadows creates a sense of visual disconnect, hindering deep emotional resonance and failing to fully realize the educational and dissemination functions of the VR experience. Summary of the Invention

[0005] This invention proposes a method for large-scale dynamic reconstruction and virtual-real collaborative rendering of digital twins for smart cultural tourism, in order to solve the problems mentioned in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for large-scale dynamic reconstruction and virtual-real collaborative rendering of digital twins for smart cultural tourism, comprising the following steps: S1. Data collection is carried out on the target cultural and tourism scene using UAV oblique photography and ground-based LiDAR scanning. The UAV is used to perform panoramic scanning at the preset first flight altitude to obtain three-dimensional point clouds and low-resolution textures. Ground-based LiDAR is used to perform close-range scanning of key buildings and cultural relics to obtain high-density point clouds and high-resolution texture images. The acquired point cloud data and texture images are spatiotemporally registered and fused to establish a two-way mapping relationship between point clouds and texture pixels and construct an initial three-dimensional geometric model of the scene. S2. A continuous implicit representation model for constructing a scene based on neural radiation field is constructed. Multi-view images and corresponding camera poses are used as inputs. The volume density of spatial points in the scene and the color information related to the viewpoint are learned through a multilayer perceptron network with jump connections. The scene image under any new viewpoint is synthesized by integrating along the ray using the volume rendering equation, thus completing the neural rendering reconstruction of the cultural tourism scene. S3. Perform real-time motion detection and semantic segmentation on dynamic elements in the scene, separate the detected dynamic foreground from the static background, use a spatiotemporal trajectory prediction model based on graph convolutional network to predict the motion trend of the dynamic foreground and update the 3D bounding box of the foreground in real time, and maintain the real-time consistency between the geometric model and the physical scene of the static background through an incremental point cloud update strategy. S4. Construct a cloud-based collaborative virtual-real fusion rendering pipeline, deploy global scene rendering of neural radiation fields on cloud servers, deploy local dynamic foreground rendering and visitor interaction responses on edge nodes, preload scene blocks corresponding to potential visitor viewpoints to edge node caches through viewpoint prediction algorithms, perform image compositing and display operations on mobile terminals, and complete the splicing and synchronous output of cloud rendering results and local terminal rendering content.

[0007] Furthermore, it also includes: after constructing the initial 3D geometric model of the scene, a semantic enhancement mechanism is introduced. A pre-trained visual semantic segmentation network is used to perform pixel-level semantic annotation on multi-view texture images. The semantic labels of building walls, ground paving, vegetation cover, exhibition facilities and tour paths are back-projected onto the surface of the 3D geometric model to establish a layered scene representation model with semantic information. In the subsequent neural radiation field rendering, the rendering accuracy and level of detail of different semantic layers are dynamically adjusted according to the areas of visitor attention. Higher rendering resolution is allocated to the semantic layers of the areas of visitor attention, and the resolution is reduced for non-visual areas.

[0008] Furthermore, it also includes: embedding a lighting consistency optimization module into the virtual-real fusion rendering pipeline, collecting ambient lighting parameters of the target cultural tourism scene in real time, and using the collected ambient lighting parameters as constraints to input the volume rendering equation of the neural radiation field, so that the virtual perspective image rendered in the cloud and the real scene image collected by the terminal maintain consistency in color and shadow direction.

[0009] Furthermore, in step S2, the volume rendering equation for the neural radiation field calculates the pixel color value corresponding to each ray by numerically integrating the scene volume density and color along the camera ray direction. The integration process uniformly samples the ray into several sampling points. The spatial coordinates of each sampling point are encoded and input into a multilayer perceptron network to obtain the volume density value and color vector of that point. Then, the final pixel color is obtained by accumulating the transmittance along the ray direction and the color of each sampling point using the following formula: ; in, For camera rays The corresponding pixel color vector; This represents the total number of sampling points along the ray; For the ray to propagate from the starting point to the... Cumulative transmittance at each sampling point For the first Volume opacity at each sampling point For the first The color vector at each sampling point.

[0010] Furthermore, the incremental point cloud update strategy for the static background in step S3 is as follows: After the initial 3D geometric model is constructed, new point cloud frames are continuously collected for the static areas in the scene. The newly collected point cloud frames are iteratively registered with the existing global point cloud map for nearest point. After registration, the point clouds in the overlapping areas are updated by position fusion using the weighted moving average method defined in the following formula: ; in, For the first The updated 3D coordinate vector after fusing the overlapping points; For the first The three-dimensional coordinate vectors of the overlapping points in the newly acquired cloud frame; For the first The three-dimensional coordinate vectors of the overlapping points in the existing global point cloud map; For the first The fusion weight coefficient of the overlapping points.

[0011] Furthermore, in step S4, the viewpoint prediction algorithm uses a long short-term memory network to predict the viewpoint position and gaze direction of the tourist in the next rendering frame based on the tourist's head movement trajectory sequence within a preset time period before the current moment. The prediction result is used to determine the set of scene block indexes preloaded in the cloud. The scene blocks are organized in a spatial octree structure and are layered according to distance from near to far with the viewpoint prediction position as the center. Scene blocks in the viewpoint directly in front of the viewpoint and adjacent viewpoints on both sides in the horizontal direction are loaded first, and scene blocks in frequently revisited areas in the tourist's historical viewpoint trajectory are cached in advance.

[0012] Furthermore, in step S4, the dynamic foreground rendering of edge nodes adopts a hybrid rendering pipeline strategy. For near-field dynamic elements that are less than the first distance threshold from the viewer's point of view, a physically based material rendering model is used for high-precision rendering. For mid-field dynamic elements that are between the first and second distance thresholds, a simplified diffuse shading model is used for medium-precision rendering. For far-field dynamic elements that are greater than the second distance threshold, a texture mapping-based patch model is used for low-precision rendering.

[0013] Furthermore, a large-scale dynamic reconstruction and virtual-real collaborative rendering system for digital twins in smart cultural tourism includes the following modules: Multi-source data acquisition module: used to interface with UAV oblique photography equipment and ground lidar scanning equipment to complete multi-scale and multi-source data acquisition of target cultural and tourism scenes, perform spatiotemporal registration and fusion of acquired point cloud data and texture images, establish a bidirectional mapping relationship between point cloud and texture pixels, and construct an initial three-dimensional geometric model of the scene with high-resolution texture. Neural Implicit Reconstruction Module: Used to construct a continuous implicit representation model of the scene based on the neural radiation field. It takes multi-view images and camera poses as input, learns the volume density and view-related color information of the scene spatial points through a multilayer perceptron network with skip connections, and uses the volume rendering equation to synthesize the scene image under any new view along the ray integral to complete the neural rendering reconstruction of the cultural tourism scene. Dynamic scene segmentation and update module: used to perform real-time motion detection and semantic segmentation of dynamic elements in the scene, separate the detected dynamic foreground from the static background, use a spatiotemporal trajectory prediction model based on graph convolutional network to predict the motion trend of the dynamic foreground and update the 3D bounding box of the foreground in real time, and use an incremental point cloud update strategy to maintain the real-time consistency between the geometric model and the physical scene for the static background. Cloud-edge-device collaborative rendering module: used to build a cloud-based collaborative virtual-real rendering pipeline, deploying global scene rendering of neural radiation fields on cloud servers, deploying local dynamic foreground rendering and visitor interaction responses on edge nodes, preloading scene blocks of potential visitor perspectives to edge node caches through viewpoint prediction algorithms, controlling mobile terminals to perform image compositing and display operations, and completing the splicing and synchronous output of cloud rendering results and terminal local rendering content.

[0014] Furthermore, the multilayer perceptron network in the neural implicit reconstruction module adopts a fully connected residual structure with skip connections. The network input consists of three-dimensional spatial coordinates and two-dimensional viewpoint direction vectors after sinusoidal position encoding. The position encoding function maps the input coordinates to a high-dimensional Fourier feature space. The network output includes a volume density scalar of spatial points and a three-dimensional color vector, where the volume density depends only on the spatial coordinates, and the color vector depends on both the spatial coordinates and the viewpoint direction.

[0015] Furthermore, when the multi-source data acquisition module performs multi-scale multi-source data acquisition, it controls the UAV to perform oblique photography at a preset first flight altitude to obtain a large-scale sparse point cloud and low-resolution texture, and controls the ground lidar to perform a preset second scanning resolution to perform a close-up scan of key buildings and cultural relics to obtain a high-density point cloud. The panoramic scan data and the close-up data are aligned and fused through a two-step registration strategy based on feature point matching and iterative nearest point. After fusion, the data is organized into a multi-resolution level scene 3D geometric model according to a spatial octree structure. The model data of different resolution levels are stored in the cloud and cached to edge nodes through a content delivery network.

[0016] Compared with existing technologies, the beneficial effects of this invention are: This invention employs a multi-source data acquisition method combining UAV oblique photography and ground-based LiDAR. This approach simultaneously balances the efficiency of acquiring data across large-scale cultural and tourism scenes with the textural accuracy of key buildings and cultural relics. Combined with implicit modeling using neural radiation fields, it can generate high-fidelity scene images from any new perspective, meeting the needs of tourists for free exploration and avoiding the difficulty of balancing accuracy and cost in traditional reconstruction methods. Furthermore, the semantic enhancement mechanism introduced on this basis dynamically adjusts the rendering precision of different semantic layers based on the areas of interest to tourists. While ensuring the rendering quality of the areas viewed by tourists, it rationally optimizes the allocation efficiency of cloud computing resources and reduces unnecessary computational consumption.

[0017] This invention separates the dynamic foreground from the static background through real-time detection and segmentation of dynamic elements. The dynamic foreground uses a trajectory prediction model to update the 3D bounding box in advance, while the static background uses an incremental point cloud update strategy to synchronize with changes in the physical scene. This maintains real-time consistency between the digital twin model and the real scene, avoiding the problem of model content becoming disconnected from the actual scene. Furthermore, the added lighting consistency optimization module adjusts the rendering effect based on real-time collected ambient lighting parameters, ensuring that the colors and shadow directions of the virtual content are consistent with the real scene, eliminating the visual disconnect between virtual and real elements and enhancing the visitor's immersive experience.

[0018] This invention employs a cloud-edge-device collaborative rendering pipeline, deploying computationally intensive global scene rendering in the cloud and low-latency dynamic foreground rendering and interactive responses on edge nodes. The terminal performs only lightweight image compositing and display operations, adapting to the deployment needs of cultural and tourism scenic areas with varying hardware conditions. This eliminates the need for deploying numerous high-performance rendering devices within the scenic area, reducing the cost of digital transformation. Combined with a viewpoint prediction algorithm that pre-loads scene blocks corresponding to potential visitor perspectives, network transmission latency is effectively reduced, preventing screen stuttering when visitors rotate their viewpoints. The layered precision rendering strategy used at edge nodes reduces the computational load on these nodes while maintaining rendering quality, supporting concurrent access from multiple visitors. Furthermore, this solution supports various application scenarios such as immersive guided tours, virtual interactive exhibitions, and immersive cultural experiences, catering to the needs of visitors of different ages and contributing to the enhanced dissemination of cultural and tourism resources. Attached Figure Description

[0019] Figure 1 This is a flowchart of the overall process of dynamic reconstruction of large scenes and virtual-real collaborative rendering for digital twins of smart cultural tourism proposed in this invention. Figure 2 This is a flowchart of the multi-source data acquisition and semantic hierarchical scene reconstruction process of the present invention; Figure 3 This is a flowchart of the dynamic and static scene separation and real-time incremental update process of the present invention; Figure 4 This is a flowchart of the cloud-edge-device collaborative virtual-real fusion rendering and viewpoint prediction process of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0022] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. Furthermore, the terms "installed," "connected," and "linked" should be interpreted broadly; for example, they may refer to a fixed connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. The invention will now be described in further detail with reference to the accompanying drawings.

[0023] Reference Figures 1 to 4 A method for large-scale dynamic reconstruction and virtual-real collaborative rendering of digital twins for smart cultural tourism includes the following steps: S1. Multi-scale, multi-source data collection is carried out on the target cultural and tourism scene by combining UAV oblique photography and ground-based LiDAR scanning. The UAV performs a panoramic scan at the preset first flight altitude to obtain a large-scale 3D point cloud and low-resolution texture. The ground-based LiDAR performs a close-range fine scan on key buildings and cultural relics to obtain high-density point cloud and high-resolution texture images. The acquired point cloud data and texture images are spatiotemporally registered and fused, and a bidirectional mapping relationship is established between point cloud and texture pixels to construct an initial 3D geometric model of the scene with high-resolution texture. S2. A continuous implicit representation model for constructing a scene based on neural radiation field. Using multi-view images and corresponding camera poses as input, the model learns the volume density and view-related color information of each spatial point in the scene through a multilayer perceptron network with jump connections. The model uses the volume rendering equation to synthesize scene images under any new view along the ray integral, thereby realizing high-fidelity neural rendering reconstruction of cultural and tourism scenes from continuous view. S3. Real-time motion detection and semantic segmentation are performed on dynamic elements in the scene, including pedestrians, tourists, exhibits, and changes in natural light and shadow. The detected dynamic foreground is separated from the static background. A spatiotemporal trajectory prediction model based on graph convolutional networks is used to predict the future motion trend of the dynamic foreground and update the 3D bounding box of the foreground in real time. An incremental point cloud update strategy is used to maintain the real-time consistency between the geometric model and the physical scene for the static background. S4. Construct a cloud-based collaborative virtual-real fusion rendering pipeline. Deploy the computationally intensive neural radiation field global scene rendering on the cloud server, and deploy the low-latency local dynamic foreground rendering and visitor interaction response on the nearest edge node. Preload the scene blocks corresponding to the potential viewpoint of the visitor to the edge node cache through the viewpoint prediction algorithm. Only perform lightweight image compositing and display operations on the mobile terminal to achieve seamless stitching and synchronous output of cloud rendering results and terminal local rendering content under different lighting conditions and dynamic scenes.

[0024] This invention also includes: introducing a semantic enhancement mechanism after constructing the initial three-dimensional geometric model of the scene; using a pre-trained visual semantic segmentation network to perform pixel-level semantic annotation on multi-view texture images; back-projecting the semantic labels of building walls, ground paving, vegetation cover, exhibition facilities and tour paths onto the surface of the three-dimensional geometric model; establishing a layered scene representation model with semantic information; so that during subsequent neural radiation field rendering, the rendering accuracy and detail level allocation of different semantic layers can be dynamically adjusted according to the areas of interest of tourists; allocating higher rendering resolution to the semantic layers of the areas of interest of tourists and appropriately reducing the resolution of non-interested areas to optimize the allocation efficiency of cloud rendering computing resources.

[0025] This invention also includes: embedding a lighting consistency optimization module in the virtual-real fusion rendering pipeline, which collects the ambient lighting parameters of the target cultural and tourism scene in real time, including the sun's azimuth and elevation angles, sky color temperature, and ambient light intensity. The collected ambient lighting parameters are used as external constraints and input into the volume rendering equation of the neural radiation field, so that the virtual perspective image generated by cloud rendering and the real scene image collected by the terminal maintain the consistency of color and shadow direction under the current ambient lighting conditions, thereby eliminating the visual disconnect between virtual and real images caused by changes in scene lighting.

[0026] In this invention, the volume rendering equation for the neural radiation field in step S2 calculates the pixel color value corresponding to each ray by numerically integrating the scene volume density and color along the camera ray direction. The integration process uniformly samples the ray into several sampling points. The spatial coordinates of each sampling point are encoded and input into a multilayer perceptron network to obtain the volume density value and color vector of that point. The final pixel color is then obtained by accumulating the transmittance along the ray direction and the color of each sampling point using the following formula: ; in, For camera rays The corresponding pixel color vector; This represents the total number of sampling points along the ray; For the ray to propagate from the starting point to the... The cumulative transmittance at each sampling point represents the proportion of light that is not blocked before reaching that point; For the first The volume opacity at each sampling point is calculated from the volume density and the sampling interval, and the value ranges from zero to one. For the first The color vector at each sampling point contains three color channel values: red, green, and blue. The volume density and color of each sampling point in the neural radiation field are accumulated and synthesized along the ray direction using the above volume rendering equation, thereby obtaining the accurate pixel color estimate corresponding to the current camera ray.

[0027] In this invention, the incremental point cloud update strategy for the static background in step S3 is as follows: After the initial 3D geometric model is constructed, new point cloud frames are continuously collected for the static areas in the scene. The newly collected point cloud frames are iteratively registered with the existing global point cloud map for nearest point. After registration, the point clouds in the overlapping areas are updated by position fusion using the weighted moving average method defined in the following formula: ; in For the first The updated 3D coordinate vector after fusing the overlapping points; For the first The three-dimensional coordinate vectors of the overlapping points in the newly acquired cloud frame; For the first The three-dimensional coordinate vectors of the overlapping points in the existing global point cloud map; For the first The fusion weight coefficient of each overlapping point is determined based on the local neighborhood consistency of the point in the old and new point clouds and the value ranges from zero to one. New point clouds in non-overlapping areas are directly added to the global point cloud map. At the same time, the confidence weight of expired point clouds is gradually reduced according to the timestamp of the new point cloud and the preset decay period, so as to gradually eliminate old point cloud data generated by natural aging or repair of the scene.

[0028] In this invention, the viewpoint prediction algorithm in step S4 is based on the sequence of tourists' head movement trajectories within a preset time period before the current moment. It uses a long short-term memory network to predict the viewpoint position and line of sight of the tourists at the next rendering frame. The prediction results are used to determine in advance the set of scene block indexes that need to be preloaded from the cloud. The scene blocks are organized in a spatial octree structure and are layered according to distance from near to far with the viewpoint prediction position as the center. Scene blocks in front of the viewpoint and adjacent viewpoints on both sides in the horizontal direction are loaded first to cover the tourists' free viewpoint rotation needs. At the same time, scene blocks in frequently revisited areas in the tourists' historical viewpoint trajectory are cached in advance to reduce repeated loading delay and network bandwidth consumption.

[0029] In this invention, the dynamic foreground rendering of edge nodes in step S4 adopts a strategy based on a hybrid rendering pipeline. For near-field dynamic elements that are less than a first distance threshold from the visitor's viewpoint, a physically based material rendering model is used for high-precision rendering. For mid-field dynamic elements that are between the first and second distance thresholds, a simplified diffuse shading model is used for medium-precision rendering. For far-field dynamic elements that are greater than the second distance threshold, a texture mapping-based patch model is used for low-precision rendering. Through this three-layer precision grading strategy, the overall rendering computation load and graphics processor memory usage of edge nodes are effectively reduced while ensuring the rendering quality of the neighborhood of the visitor's viewpoint. This allows lightweight edge nodes to also undertake dynamic foreground rendering tasks for multiple concurrent visitors.

[0030] This invention includes a multi-source data acquisition module, a neural implicit reconstruction module, a dynamic scene segmentation and update module, and a cloud-edge-device collaborative rendering module. The multi-source data acquisition module is used to interface with UAV oblique photography equipment and ground lidar scanning equipment to complete multi-scale multi-source data acquisition of target cultural and tourism scenes. It performs spatiotemporal registration and fusion of the acquired point cloud data and texture images, establishes a bidirectional mapping relationship between point cloud and texture pixels, and constructs an initial three-dimensional geometric model of the scene with high-resolution texture. The neural implicit reconstruction module is used to construct a continuous implicit representation model of the scene based on the neural radiation field. It takes multi-view images and corresponding camera poses as input, learns the volume density and view-related color information of each spatial point in the scene through a multilayer perceptron network with jump connections, and synthesizes scene images under any new view by integrating along the light rays using the volume rendering equation, so as to realize continuous view high-fidelity neural rendering reconstruction of cultural and tourism scenes. The dynamic scene segmentation and update module is used to perform real-time motion detection and semantic segmentation on dynamic elements in the scene, including pedestrians, tourists, exhibition props and natural light and shadow changes. It separates the detected dynamic foreground from the static background, uses a spatiotemporal trajectory prediction model based on graph convolutional networks to predict the future motion trend of the dynamic foreground and updates the 3D bounding box of the foreground in real time, and maintains the real-time consistency between the geometric model and the physical scene through an incremental point cloud update strategy for the static background. The cloud-edge-device collaborative rendering module is used to construct a cloud-based collaborative virtual-real fusion rendering pipeline. It deploys the computationally intensive neural radiation field global scene rendering on the cloud server and deploys the low-latency local dynamic foreground rendering and visitor interaction response on the nearest edge node. It preloads scene blocks corresponding to the potential viewpoint of the visitor into the edge node cache through the viewpoint prediction algorithm and controls the mobile terminal to perform only lightweight image compositing and display operations. This enables seamless stitching and synchronous output of cloud rendering results and terminal local rendering content under different lighting conditions and dynamic scenes.

[0031] In this invention, the multilayer perceptron network in the neural implicit reconstruction module adopts a fully connected residual structure with skip connections. The network input consists of three-dimensional spatial coordinates and two-dimensional viewpoint direction vectors after sinusoidal position encoding. The position encoding function maps the input coordinates to a high-dimensional Fourier feature space to enhance the network's ability to fit high-frequency geometric details. The network output includes a volume density scalar and a color three-dimensional vector corresponding to the spatial point. The volume density depends only on the spatial coordinates, while the color vector depends on both the spatial coordinates and the viewpoint direction to model the viewpoint-related diffuse reflection and specular lighting effects in cultural and tourism scenes.

[0032] In this invention, when the multi-source data acquisition module performs multi-scale multi-source data acquisition, it controls the UAV to perform a panoramic scan of the scene at a preset first flight altitude to obtain a large-scale sparse point cloud and low-resolution texture, and controls the ground-based lidar to perform a close-range fine scan of key buildings and cultural relics at a preset second scanning resolution to obtain a high-density point cloud. The panoramic scan data and the close-range fine data are aligned and fused through a two-step registration strategy based on feature point matching and iterative nearest point. After fusion, the data is organized into a multi-resolution hierarchical progressively loadable 3D geometric model of the scene according to a spatial octree structure. The model data of different resolution levels are stored in the cloud and cached to the edge node closest to the target cultural tourism scene through a content delivery network.

[0033] The specific embodiments of the present invention are further illustrated below: Example 1: Digital Twin Reconstruction and Virtual-Real Guided Tour of Core Cultural Tourism Scenic Areas This embodiment is applied to a core cultural and tourism scenic area, implementing technology to meet the immersive guided tour needs of the entire scenic area. First, multi-scale, multi-source data acquisition is conducted. A drone is controlled to perform a panoramic scan of the entire scenic area at a preset initial flight altitude, acquiring large-scale 3D point clouds and low-resolution texture images. Simultaneously, a ground-based LiDAR is used to perform close-range, detailed scanning of key historical sites, memorial exhibition locations, and other important cultural relics, acquiring high-density point clouds and high-resolution texture images. After acquisition, feature point matching and iterative nearest-point algorithms are used to complete the spatiotemporal registration of the two types of data. After alignment, a bidirectional mapping relationship is established between the 3D coordinates of the point cloud and the pixel coordinates of the texture, constructing an initial 3D geometric model of the scene with high-resolution textures.

[0034] After the initial model construction is completed, a semantic enhancement mechanism is introduced. A pre-trained visual semantic segmentation network is used to perform pixel-level semantic annotation on all multi-view texture images, identifying five semantic objects: building walls, ground paving, vegetation cover, exhibition facilities, and visitor paths. The obtained semantic labels are then back-projected onto the surface of the 3D geometric model to establish a hierarchical scene representation model with semantic information. During subsequent rendering, the eye-tracking module on the terminal identifies the visitor's gaze area in real time, allocating higher rendering resolution to the semantic layer corresponding to the gaze area and appropriately reducing the resolution for non-gaze areas, thereby optimizing the allocation efficiency of cloud rendering computing resources.

[0035] Subsequently, a continuous implicit representation model of the scene is constructed based on the neural radiation field. Using acquired multi-view images and corresponding camera poses as input, a multilayer perceptron network with skip connections is employed to learn the volume density and view-dependent color information of each spatial point in the scene. This multilayer perceptron uses an 8-layer fully connected structure. The first 5 layers receive position-encoded 3D spatial coordinate features and output the volume density scalar of the corresponding spatial point. The position-encoded features of the first layer are connected to the output of the fifth layer via skip connections to avoid the loss of high-frequency geometric details. The last 3 layers combine the input 2D view direction vector to output red, green, and blue color vectors for the corresponding spatial point. The volume density depends only on the spatial coordinates, while the color vectors depend on both the spatial coordinates and the view direction, thus modeling the diffuse and specular lighting effects in the scene. After training, scene images under arbitrary new viewpoints are synthesized by integrating along the ray using the volume rendering equation, achieving high-fidelity neural rendering reconstruction of the scenic area from continuous viewpoints.

[0036] Next, real-time processing of dynamic elements is carried out. Real-time motion detection and semantic segmentation are performed on pedestrians, tourists, moving exhibits, and changes in natural light and shadow within the scenic area to separate the detected dynamic foreground from the static background. For the dynamic foreground, a spatiotemporal trajectory prediction model based on graph convolutional networks is used. The model takes the target position features and semantic features of 10 consecutive frames as input and outputs the motion trend of the next 3 frames, thus updating the foreground's 3D bounding box in real time. For the static background, an incremental point cloud update strategy is adopted. New point cloud frames are continuously collected from static areas, and these frames are registered with the existing global point cloud map. Point clouds in overlapping areas are updated using a weighted moving average method, while newly added point clouds in non-overlapping areas are directly appended to the global point cloud map. Simultaneously, the confidence weight of expired point clouds is gradually reduced according to a preset decay period, progressively eliminating outdated point cloud data generated by natural scene aging and facility repairs, maintaining real-time consistency between the geometric model and the physical scene.

[0037] Subsequently, a cloud-edge-device collaborative virtual-real fusion rendering pipeline was constructed. The computationally intensive neural radiation field global scene rendering was deployed to the cloud server, while the low-latency local dynamic foreground rendering and visitor interaction responses were deployed to edge nodes at the scenic area entrance. A viewpoint prediction algorithm was used to preload scene blocks corresponding to potential visitor perspectives. This algorithm, based on the visitor's head movement trajectory sequence within the previous 5 seconds, used a Long Short-Term Memory network to predict the viewpoint position and gaze direction at the next rendering frame, thus determining the set of scene block indices to be preloaded. Scene blocks were organized in a spatial octree structure, with the predicted viewpoint position as the center and loading priority set hierarchically from near to far. Scene blocks from adjacent perspectives directly in front of the viewpoint and on both sides in the horizontal direction were loaded first. Simultaneously, scene blocks from frequently revisited areas in the visitor's historical viewpoint trajectory were pre-cached to reduce repetitive loading latency and network bandwidth consumption. The mobile terminal only performed lightweight image compositing and display operations, achieving seamless stitching and synchronous output of cloud rendering results and local terminal rendering content.

[0038] An embedded lighting consistency optimization module is used during the rendering process. Lighting acquisition devices are uniformly deployed throughout the scenic area to collect real-time parameters such as solar azimuth angle, altitude angle, sky color temperature, and ambient light intensity. These parameters are then used as external constraints input into the volume rendering equation of the neural radiation field to adjust the lighting parameters of the virtual content. This ensures that the virtual viewpoint image generated by cloud rendering maintains color and shadow direction consistency with the real-world image captured by the terminal under the current ambient lighting conditions, eliminating the visual disconnect between virtual and real images caused by changes in scene lighting. Dynamic foreground rendering of edge nodes employs a hybrid rendering pipeline strategy. Near-field dynamic elements within a first distance threshold from the viewer's perspective are rendered with high precision using a physically based material rendering model. Mid-field dynamic elements between the first and second distance thresholds are rendered with medium precision using a simplified diffuse shading model. Distant dynamic elements beyond the second distance threshold are rendered with low precision using a texture mapping-based patch model. This reduces the overall rendering computational load and GPU memory usage of edge nodes. An adaptive task allocation strategy based on dynamic evaluation of rendering load is adopted between the cloud and edge nodes. The cloud server continuously monitors the current rendering queue length, graphics processor utilization and network transmission latency of each edge node. When the rendering load of an edge node exceeds the preset load threshold, some dynamic foreground rendering tasks in the rendering queue of that edge node are migrated to the cloud or adjacent low-load edge nodes for execution. During the migration process, intermediate transition frames are generated through inter-frame motion vector interpolation to avoid screen stuttering or frame rate drop caused by dynamic switching of rendering nodes.

[0039] This embodiment effectively solves the problems of weak interactivity and insufficient digital coverage in the original static exhibitions of cultural and tourism scenic spots. The multi-source data acquisition method takes into account both the modeling efficiency of the entire scenic area and the texture accuracy of key cultural relics. The neural radiation field modeling supports high-fidelity rendering from any perspective, meeting the needs of tourists for free exploration. The cloud-edge-device collaborative rendering architecture does not require tourists to have high-performance terminals. Ordinary smartphones can realize immersive virtual and real guided tours. Semantic layered rendering and dynamic task scheduling strategies effectively optimize computing power allocation, support multiple tourists to access at the same time, and the consistent design of virtual and real lighting enhances the immersive experience, allowing tourists to more intuitively feel the profound connotations, while reducing the digital transformation cost and operation and maintenance pressure of scenic spots.

[0040] Example 2: Integrated Virtual and Real Display and Cloud Tour Service for Themed Memorial Venues This embodiment is applied to a themed memorial venue, implementing technology to meet the needs of immersive offline venue displays and online cloud tour services. First, multi-scale, multi-source data acquisition is conducted. A drone is controlled to perform a panoramic scan of the memorial venue and surrounding outdoor historical sites at a preset first flight altitude, acquiring large-scale sparse point clouds and low-resolution texture images. Simultaneously, a ground-based LiDAR is used to perform close-range, detailed scanning of key areas within the memorial venue, such as cultural relics, commemorative landmarks, and core historical scenes, acquiring high-density point clouds and high-resolution texture images at a preset second scan resolution. After acquisition, a two-step registration strategy using feature point matching and iterative nearest point is used to align and fuse the two types of data, establishing a bidirectional mapping relationship between point cloud 3D coordinates and texture pixel coordinates. An initial 3D geometric model of the scene with high-resolution textures is constructed. The fused model is organized into a multi-resolution, progressively loadable 3D scene model using a spatial octree structure. Model data at different resolution levels is stored in the cloud and cached to the nearest edge node to the venue via a content delivery network.

[0041] After the initial model construction is completed, a semantic enhancement mechanism is introduced. A pre-trained visual semantic segmentation network is used to perform pixel-level semantic annotation on all multi-view texture images, identifying five semantic objects: commemorative buildings, historical sites, exhibition facilities, vegetation cover, and visitor paths. The obtained semantic labels are then back-projected onto the surface of the 3D geometric model to establish a hierarchical scene representation model with semantic information. In the subsequent rendering process, areas of interest are identified based on visitor interactions. Higher rendering resolution is allocated to the semantic layers corresponding to areas that visitors click on, query, or linger in, while the resolution is appropriately reduced for non-interested areas, thereby optimizing the allocation efficiency of cloud rendering computing resources.

[0042] Subsequently, a continuous implicit representation model of the scene is constructed based on the neural radiation field. Using acquired multi-view images and corresponding camera poses as input, a multilayer perceptron network with a fully connected residual structure and skip connections is employed to learn the volume density and view-dependent color information of each spatial point in the scene. The input 3D spatial coordinates and 2D view direction vectors are first mapped to a high-dimensional Fourier feature space through sinusoidal position encoding, enhancing the network's ability to fit high-frequency geometric details. The network outputs a volume density scalar and a 3D color vector for the corresponding spatial point. The volume density depends only on the spatial coordinates, while the color vector depends on both the spatial coordinates and the view direction, thus modeling the view-dependent diffuse reflection and specular lighting effects in the scene. After training, scene images under arbitrary new viewpoints are synthesized by integrating along the ray using the volume rendering equation, achieving continuous viewpoint high-fidelity neural rendering reconstruction of the memorial hall and heritage areas. This allows online visitors to freely adjust their viewpoints to view the entire historical scene.

[0043] Next, real-time processing of dynamic elements is carried out. Real-time motion detection and semantic segmentation are performed on pedestrians and visitors inside the museum, moving exhibits, and changes in natural light and shadow outdoors, separating the detected dynamic foreground from the static background. For the dynamic foreground, a spatiotemporal trajectory prediction model based on graph convolutional networks is used. The model takes the target position features and semantic features of 10 consecutive frames as input and outputs the motion trend of the next 3 frames, thus updating the foreground's 3D bounding box in real time. For the static background, an incremental point cloud update strategy is adopted. New point cloud frames from static areas are continuously collected, and these frames are registered with the existing global point cloud map. Point clouds in overlapping areas are updated using a weighted moving average method, while newly added point clouds from non-overlapping areas are directly appended to the global point cloud map. Simultaneously, the confidence weight of expired point clouds is gradually reduced according to a preset decay period, progressively eliminating outdated point cloud data generated by site restoration and natural vegetation growth, maintaining real-time consistency between the geometric model and the physical scene.

[0044] Subsequently, a cloud-edge-device collaborative virtual-real fusion rendering pipeline was constructed. The computationally intensive global historical scene rendering of the neural radiation field was deployed to the cloud server, while the low-latency local dynamic foreground rendering and visitor interaction responses were deployed to edge nodes within the memorial hall's server room. A viewpoint prediction algorithm was used to preload scene blocks corresponding to potential visitor perspectives. This algorithm, based on the visitor's head movement trajectory or terminal swiping sequence within the previous 5 seconds, used a Long Short-Term Memory network to predict the viewpoint position and viewing direction at the next rendering frame, thus determining the set of scene block indices to be preloaded. Scene blocks were organized in a spatial octree structure, with the predicted viewpoint position as the center and loading priority set hierarchically from near to far, prioritizing scene blocks directly in front of the viewpoint and adjacent viewpoints on both sides in the horizontal direction. Simultaneously, scene blocks in frequently revisited areas from the visitor's historical viewpoint trajectory were pre-cached to reduce repetitive loading latency and network bandwidth consumption. Touchscreen terminals in the physical museum, visitor mobile terminals, and online cloud-based user terminals only performed lightweight image compositing and display operations, achieving seamless stitching and synchronous output of cloud-rendered results and locally captured real-world content.

[0045] An embedded lighting consistency optimization module is used during the rendering process. Lighting acquisition devices are evenly deployed in both indoor and outdoor heritage areas to collect real-time parameters such as solar azimuth angle, altitude angle, sky color temperature, and ambient light intensity. These parameters are then used as external constraints input into the volume rendering equation of the neural radiation field to adjust the lighting parameters of the virtual content. This ensures that the virtual historical scenes and figures generated by cloud rendering maintain color and shadow direction consistency with the real-world images captured by the terminal under the current ambient lighting conditions, eliminating the visual disconnect between virtual and real images caused by changes in scene lighting. Dynamic foreground rendering of edge nodes employs a hybrid rendering pipeline strategy. Near-field dynamic elements within a first distance threshold from the visitor's viewpoint are rendered with high precision using a physically based material rendering model. Mid-field dynamic elements between the first and second distance thresholds are rendered with medium precision using a simplified diffuse shading model. Distant dynamic elements beyond the second distance threshold are rendered with low precision using a texture mapping-based patch model. This reduces the overall rendering computational load and GPU memory usage of edge nodes. An adaptive task allocation strategy based on dynamic evaluation of rendering load is adopted between the cloud and edge nodes. The cloud server continuously monitors the current rendering queue length, graphics processor utilization and network transmission latency of each edge node. When the rendering load of a certain edge node exceeds the preset load threshold during peak holiday travel periods, some non-core rendering tasks in the rendering queue of that edge node are migrated to the cloud or adjacent low-load edge nodes for execution. During the migration process, intermediate transition frames are generated through inter-frame motion vector interpolation to avoid screen stuttering or frame rate drop caused by dynamic switching of rendering nodes.

[0046] This embodiment effectively solves the problems of fixed perspectives in online cloud tours and insufficient immersion in offline exhibitions in traditional themed memorial venues. The multi-source data acquisition method balances the efficiency of large-scale modeling of outdoor relics with the detail precision of key artifacts. Neural radiation field modeling supports free browsing from any perspective, allowing online visitors to view the historical scene layout from all angles, while offline visitors can intuitively understand the historical events through the fusion of virtual and real content. The cloud-edge-device collaborative rendering architecture ensures low-latency interaction and eliminates screen stuttering issues. The consistent design of virtual and real lighting avoids visual disjointedness, effectively enhancing the visitor's immersive experience while reducing the digital operation and maintenance costs of memorial venues. It can support a large number of simultaneous online visitors.

[0047] Reference Figure 1This flowchart illustrates the overall execution architecture of a method for large-scale dynamic reconstruction and virtual-real collaborative rendering of digital twins for smart cultural tourism. First, the system uses a drone and LiDAR to collaboratively acquire multi-scale, multi-source spatial data and construct an initial 3D geometric model with high-resolution textures. Then, based on multi-view images, a continuous implicit representation model of neural radiation fields is built to achieve high-fidelity scene perspective synthesis. On this basis, the system performs semantic segmentation of the scene, separating the dynamic foreground from the static background, and implementing motion trend prediction and incremental point cloud updates respectively to maintain real-time consistency of the physical space. Finally, by constructing a cloud-edge-device collaborative rendering pipeline and rationally allocating computational load, the system achieves efficient synchronous synthesis of virtual and real images.

[0048] Reference Figure 2 This flowchart details the specific steps of multi-source data acquisition and semantic-level 3D scene reconstruction. The system combines the advantages of large-scale panoramic scanning by UAVs and the high-precision close-range detection of ground-based LiDAR to acquire multi-resolution spatial point clouds and texture materials. Through spatiotemporal registration technology, the system establishes a bidirectional mapping between point cloud coordinates and texture pixels. To further improve rendering efficiency and scene recognition, the system introduces a visual semantic segmentation network to perform deep semantic annotation on multi-view texture images and back-project the resulting semantic labels onto the surface of the 3D geometric model. This process constructs a hierarchical scene representation model with semantic information, laying the foundation for subsequent on-demand allocation of rendering computing power and optimization of details in the visitor's viewpoint area.

[0049] Reference Figure 3 This flowchart illustrates the real-time separation and update mechanism for dynamic elements and static backgrounds in a scene. The system monitors the cultural and tourism scene in real time, accurately segmenting dynamic foreground elements such as pedestrians and exhibition props from static architectural backgrounds. For dynamic foreground elements, the system deploys a graph convolutional network to analyze their spatiotemporal trajectories, predict future movement trends, and dynamically update the 3D bounding boxes. For static backgrounds, the system employs an incremental update strategy, iteratively registering the continuously acquired new point cloud frames with the global map using nearest-neighbor registration. Overlapping area data is updated using a location fusion algorithm, while newly added non-overlapping areas are directly appended. Simultaneously, confidence weight decay processing is performed on older point clouds using timestamps to ensure that the geometric features of the digital scene remain consistent with the physical scene over the long term.

[0050] Reference Figure 4This flowchart illustrates the execution logic of the cloud-edge-device collaborative virtual-real fusion rendering pipeline. The computationally intensive neural radiation field global rendering task is deployed on the cloud server, with ambient lighting parameters fed in in real-time to ensure consistency in color and shadow direction between the virtual generated image and the real environment. Edge nodes utilize long short-term memory networks to predict the visitor's next viewpoint position and viewing direction, pre-caching high-priority spatial scene blocks and implementing multi-precision, hierarchical rendering strategies for dynamic foregrounds at different distances to reduce latency. Finally, the mobile terminal receives global background data from the cloud and dynamic foreground data from the edge, performs lightweight image compositing, and presents a seamlessly stitched, immersive tour experience to the visitor.

[0051] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for large-scale dynamic reconstruction and virtual-real collaborative rendering of digital twins for smart cultural tourism, characterized in that, Includes the following steps: S1. Data collection is carried out on the target cultural and tourism scene using UAV oblique photography and ground-based LiDAR scanning. The UAV is used to perform panoramic scanning at the preset first flight altitude to obtain three-dimensional point clouds and low-resolution textures. Ground-based LiDAR is used to perform close-range scanning of key buildings and cultural relics to obtain high-density point clouds and high-resolution texture images. The acquired point cloud data and texture images are spatiotemporally registered and fused to establish a two-way mapping relationship between point clouds and texture pixels and construct an initial three-dimensional geometric model of the scene. S2. A continuous implicit representation model for constructing a scene based on neural radiation field is constructed. Multi-view images and corresponding camera poses are used as inputs. The volume density of spatial points in the scene and the color information related to the viewpoint are learned through a multilayer perceptron network with jump connections. The scene image under any new viewpoint is synthesized by integrating along the ray using the volume rendering equation, thus completing the neural rendering reconstruction of the cultural tourism scene. S3. Perform real-time motion detection and semantic segmentation on dynamic elements in the scene, separate the detected dynamic foreground from the static background, use a spatiotemporal trajectory prediction model based on graph convolutional network to predict the motion trend of the dynamic foreground and update the 3D bounding box of the foreground in real time, and maintain the real-time consistency between the geometric model and the physical scene of the static background through an incremental point cloud update strategy. S4. Construct a cloud-based collaborative virtual-real fusion rendering pipeline, deploy global scene rendering of neural radiation fields on cloud servers, deploy local dynamic foreground rendering and visitor interaction responses on edge nodes, preload scene blocks corresponding to potential visitor viewpoints to edge node caches through viewpoint prediction algorithms, perform image compositing and display operations on mobile terminals, and complete the splicing and synchronous output of cloud rendering results and local terminal rendering content.

2. The method for large-scale dynamic reconstruction and virtual-real collaborative rendering of digital twins for smart cultural tourism as described in claim 1, characterized in that, Also includes: After constructing the initial 3D geometric model of the scene, a semantic enhancement mechanism is introduced. A pre-trained visual semantic segmentation network is used to perform pixel-level semantic annotation on multi-view texture images. The semantic labels of building walls, ground paving, vegetation cover, exhibition facilities and tour paths are back-projected onto the surface of the 3D geometric model to establish a layered scene representation model with semantic information. During subsequent neural radiation field rendering, the rendering accuracy and level of detail of different semantic layers are dynamically adjusted according to the areas of visitor attention. Higher rendering resolution is allocated to the semantic layers of the areas of visitor attention, and the resolution is reduced for non-visual areas.

3. The method for large-scale dynamic reconstruction and virtual-real collaborative rendering of digital twins for smart cultural tourism as described in claim 1, characterized in that, Also includes: An illumination consistency optimization module is embedded in the virtual-real fusion rendering pipeline to collect ambient lighting parameters of the target cultural and tourism scene in real time. The collected ambient lighting parameters are used as constraints to input the volume rendering equation of the neural radiation field, so that the virtual view image rendered in the cloud and the real scene image collected by the terminal maintain the consistency of color and shadow direction.

4. The method for large-scale dynamic reconstruction and virtual-real collaborative rendering of digital twins for smart cultural tourism as described in claim 1, characterized in that, In step S2, the volume rendering equation for the neural radiation field calculates the pixel color value corresponding to each ray by numerically integrating the scene volume density and color along the camera ray direction. The integration process uniformly samples the ray into several sampling points. The spatial coordinates of each sampling point are encoded and input into a multilayer perceptron network to obtain the volume density value and color vector of that point. Then, the final pixel color is obtained by accumulating the transmittance along the ray direction and the color of each sampling point using the following formula: ； in, For camera rays The corresponding pixel color vector; This represents the total number of sampling points along the ray; For the ray to propagate from the starting point to the... Cumulative transmittance at each sampling point For the first Volume opacity at each sampling point For the first The color vector at each sampling point.

5. The method for large-scale dynamic reconstruction and virtual-real collaborative rendering of digital twins for smart cultural tourism as described in claim 1, characterized in that, The incremental point cloud update strategy for the static background in step S3 is as follows: After the initial 3D geometric model is constructed, new point cloud frames are continuously collected for the static areas in the scene. The newly collected point cloud frames are iteratively registered with the existing global point cloud map. After registration, the point clouds in the overlapping areas are updated by position fusion using the weighted moving average method defined in the following formula: ； in, For the first The updated 3D coordinate vector after fusing the overlapping points; For the first The three-dimensional coordinate vectors of the overlapping points in the newly acquired cloud frame; For the first The three-dimensional coordinate vectors of the overlapping points in the existing global point cloud map; For the first The fusion weight coefficient of the overlapping points.

6. The method for large-scale dynamic reconstruction and virtual-real collaborative rendering of digital twins for smart cultural tourism as described in claim 1, characterized in that, In step S4, the viewpoint prediction algorithm uses a long short-term memory network to predict the viewpoint position and gaze direction of the tourist in the next rendering frame based on the tourist's head movement trajectory sequence within a preset time period before the current moment. The prediction result is used to determine the set of scene block indexes preloaded in the cloud. The scene blocks are organized in a spatial octree structure and are layered according to distance from near to far with the viewpoint prediction position as the center. Scene blocks in the viewpoint directly in front of the viewpoint and adjacent viewpoints on both sides in the horizontal direction are loaded first, and scene blocks in frequently revisited areas in the tourist's historical viewpoint trajectory are cached in advance.

7. The method for large-scale dynamic reconstruction and virtual-real collaborative rendering of digital twins for smart cultural tourism as described in claim 1, characterized in that, In step S4, the dynamic foreground rendering of edge nodes adopts a hybrid rendering pipeline strategy. For near-field dynamic elements that are less than the first distance threshold from the viewer's point of view, a physically based material rendering model is used for high-precision rendering. For mid-field dynamic elements that are between the first and second distance thresholds, a simplified diffuse shading model is used for medium-precision rendering. For far-field dynamic elements that are greater than the second distance threshold, a texture mapping-based patch model is used for low-precision rendering.

8. A large-scene dynamic reconstruction and virtual-real collaborative rendering system for smart cultural tourism digital twins, applied to the large-scene dynamic reconstruction and virtual-real collaborative rendering method for smart cultural tourism digital twins as described in any one of claims 1 to 7, characterized in that, Includes the following modules: Multi-source data acquisition module: used to interface with UAV oblique photography equipment and ground lidar scanning equipment to complete multi-scale and multi-source data acquisition of target cultural and tourism scenes, perform spatiotemporal registration and fusion of acquired point cloud data and texture images, establish a bidirectional mapping relationship between point cloud and texture pixels, and construct an initial three-dimensional geometric model of the scene with high-resolution texture. Neural Implicit Reconstruction Module: Used to construct a continuous implicit representation model of the scene based on the neural radiation field. It takes multi-view images and camera poses as input, learns the volume density and view-related color information of the scene spatial points through a multilayer perceptron network with skip connections, and uses the volume rendering equation to synthesize the scene image under any new view along the ray integral to complete the neural rendering reconstruction of the cultural tourism scene. Dynamic scene segmentation and update module: used to perform real-time motion detection and semantic segmentation of dynamic elements in the scene, separate the detected dynamic foreground from the static background, use a spatiotemporal trajectory prediction model based on graph convolutional network to predict the motion trend of the dynamic foreground and update the 3D bounding box of the foreground in real time, and use an incremental point cloud update strategy to maintain the real-time consistency between the geometric model and the physical scene for the static background. Cloud-edge-device collaborative rendering module: used to build a cloud-based collaborative virtual-real rendering pipeline, deploying global scene rendering of neural radiation fields on cloud servers, deploying local dynamic foreground rendering and visitor interaction responses on edge nodes, preloading scene blocks of potential visitor perspectives to edge node caches through viewpoint prediction algorithms, controlling mobile terminals to perform image compositing and display operations, and completing the splicing and synchronous output of cloud rendering results and terminal local rendering content.

9. The large-scene dynamic reconstruction and virtual-real collaborative rendering system for digital twins in smart cultural tourism as described in claim 8, is characterized in that, The neural implicit reconstruction module uses a multilayer perceptron network with a fully connected residual structure with skip connections. The network input consists of three-dimensional spatial coordinates and a two-dimensional viewpoint direction vector after sinusoidal position encoding. The position encoding function maps the input coordinates to a high-dimensional Fourier feature space. The network output includes a volume density scalar of spatial points and a three-dimensional color vector. The volume density depends only on the spatial coordinates, while the color vector depends on both the spatial coordinates and the viewpoint direction.

10. The large-scene dynamic reconstruction and virtual-real collaborative rendering system for digital twins in smart cultural tourism as described in claim 8, characterized in that, When the multi-source data acquisition module performs multi-scale multi-source data acquisition, it controls the UAV to perform oblique photography at a preset first flight altitude to obtain a large-scale sparse point cloud and low-resolution texture for panoramic scanning. It controls the ground-based lidar to perform close-up scanning of key buildings and cultural relics at a preset second scanning resolution to obtain high-density point clouds. The panoramic scan data and close-up data are aligned and fused through a two-step registration strategy based on feature point matching and iterative nearest point. After fusion, the data is organized into a multi-resolution hierarchical 3D geometric model of the scene according to a spatial octree structure. The model data of different resolution levels are stored in the cloud and cached to edge nodes through a content delivery network.

Images