Scenic area AR (Augmented Reality) navigation system supporting dynamic loading of 3D (Three-Dimensional) model and data processing method

Abstract

The invention discloses a scenic spot AR guide system supporting dynamic loading of a 3D model and a data processing method. According to the technical scheme of the invention, through the innovative design of a dynamic resource scheduling algorithm, a multi-level cache architecture and a hybrid rendering pipeline, the long-standing key technical problems of poor equipment adaptability, high network dependency, serious visual distortion and the like in the AR guide field are systematically solved. Practical application data shows that in typical scenes such as scenic spots, libraries and scientific halls, system performance indexes comprehensively exceed existing schemes, and reliable technical support is provided for large-scale commercialization.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of the cross of augmented reality and computer graphics, in particular to a scenic spot AR guide system supporting dynamic loading of a 3D model and a data processing method. BACKGROUND

[0002] With the rapid development of augmented reality (AR) technology, its application in cultural tourism, education display and other fields is becoming increasingly widespread. However, the existing AR guide system still has significant technical bottlenecks when dealing with large-scale complex scenes:

[0003] 1. Defects in static resource loading mechanism:

[0004] The traditional scheme (such as CN112987789A) adopts fixed resolution model loading, which causes serious lag (frame rate < 15 fps) when running on low-end devices, and high-end devices cannot fully exert their performance advantages.

[0005] The published literature "IEEE Trans. on Visualization and Computer Graphics, 2021" points out that the existing LOD (level of detail) switching algorithm will produce obvious visual jump (SSIM < 0.85) when the model simplification rate is > 70%, affecting user experience.

[0006] 2. Too high network dependence:

[0007] As described in US20220156789A1, the existing system has a model loading failure rate > 60% when the bandwidth is < 5 Mbps, and only provides a basic 2D map after network interruption, losing the core value of AR.

[0008] Scenic spots, exhibition halls and other real scenes often have network coverage blind spots (such as the deep courtyard of the Forbidden City and the underground book library of the library), resulting in frequent service interruptions.

[0009] 3. Insufficient adaptation to multi-source environment:

[0010] The comparative literature JP2022087654A discloses that the existing AR system lacks dynamic adaptation ability to device performance and environmental light, and the display difference of the same model on OLED and LCD screens is up to 42%, and the matching error of the shadow direction of the virtual object and the real scene is > 15°.

[0011] The above technical defects seriously restrict the large-scale commercial application of AR guide systems, especially in scientific museums, libraries and other scenes that require high-precision models and real-time interaction. SUMMARY

[0012] The main purpose of the present application is to provide a scenic spot AR guide system supporting 3D model dynamic loading and a data processing method, and the technical scheme of the present application solves the key technical problems such as poor device adaptability, high network dependence and serious visual distortion in the field of AR guide through the innovative design of dynamic resource scheduling algorithm, multi-level cache architecture and hybrid rendering pipeline.

[0013] To achieve the above purpose, the present application provides a scenic spot AR guide system supporting 3D model dynamic loading in one aspect, comprising the following technical modules:

[0014] High-precision data acquisition module:

[0015] RIEGL VZ-400i laser radar is used to scan the terrain of the scenic spot, and the point cloud density is greater than or equal to 200 points per square meter;

[0016] A 0.5cm resolution texture map is obtained by aerial photography of a DJI Matrice 300RTK unmanned aerial vehicle;

[0017] Blender is used to fuse and process the point cloud and the texture, and generate a glTF 2.0 format 3D model containing a normal map and an AO map;

[0018] Dynamic loading control module, comprising:

[0019] Device performance detection unit: obtain the GPU model through the OpenGL ES 3.2 interface, query the preset database to match the computing power level (level 1-10, corresponding to 1-10 TFLOPS);

[0020] Network state evaluation unit: calculate the real-time bandwidth by using a weighted moving average algorithm, and the formula is:

[0021]

[0022] When the bandwidth is lower than the threshold value 2Mbps, the degradation loading strategy is triggered;

[0023] Position sensing unit: realize sub-meter positioning accuracy by combining GPS (positioning error ≤3m) and visual SLAM (ORB-SLAM3 algorithm).

[0024] Multi-level cache architecture:

[0025] Local storage layer: cache all LOD0 level models (the number of facets is less than or equal to 100,000) within a radius of 50m of the user;

[0026] Edge node layer: deploy Huawei Atlas 500 intelligent edge device, store the full amount of LOD3 level models (the number of facets is less than or equal to 50,000) of the scenic spot, and the response time is less than 100ms;

[0027] Cloud storage layer: Aliyun OSS stores the original LOD0 level model, supports HTTPS protocol block transmission.

[0028] AR rendering engine:

[0029] Unity 2022LTS version is adopted, and AR Foundation 5.0 framework is integrated;

[0030] Light consistency processing: the scene HDR light information is collected in real time through the ambient light probe (sampling frequency 30Hz), and the diffuse reflection coefficient (range 0.6-1.2) of the virtual model is adjusted synchronously.

[0031] Preferably, the network state evaluation unit comprises a packet loss rate compensation mechanism, when the packet loss rate is detected to be greater than 5%, the key frame data is automatically switched to UDP protocol transmission (transmission interval ≤200ms).

[0032] Preferably, the edge node layer adopts a difference synchronization algorithm, and data synchronization is performed with the cloud through a JSON difference file (difference rate <15%) from 23:00 to 5:00 every day, and the synchronization bandwidth occupancy rate is ≤20%.

[0033] Another aspect of the application provides a 3D model dynamic loading scenic spot AR guide data processing method, comprising the following steps:

[0034] S1, model preprocessing

[0035] The original 3D model is optimized by LOD nine-level layering:

[0036] LOD0 level retains 100% of the number of facets (2 million facets);

[0037] The number of facets of LODn level decreases according to the formula:

[0038] N n =N n-1 ×(0.65-0.05n)(1≤n≤9)

[0039] S2, dynamic loading decision

[0040] Device adaptation strategy:

[0041] According to the GPU computing power, the LOD level is selected, and the matching rule is:

[0042]

[0043] Network adaptation strategy:

[0044] Bandwidth grading and corresponding loading scheme:

[0045]

[0046] Step S3, preloading and cache updating

[0047] Preloading radius calculation based on user moving speed:

[0048] (unit: meter, υ is the current speed in km / h)

[0049] The LRU algorithm is used to manage the local cache, and when the storage capacity exceeds 512MB, 20% of the oldest models are automatically deleted.

[0050] Preferably, in step S2, a priority promotion strategy is implemented for the gaze area model, the user gaze coordinates are recognized by an eye tracking camera (sampling rate 120Hz), and the models within the 10° cone area of the visual center are forced to load LOD0 level.

[0051] The scenic AR guide system and data processing method supporting dynamic loading of 3D models provided by the application achieve significant improvement in the following aspects:

[0052] 1. Adaptive resource scheduling optimization

[0053] Device performance matching:

[0054] Based on the dynamic LOD selection algorithm of GPU computing power, the rendering frame rate of low-end devices (such as Snapdragon 680) is improved by 300% (from 11fps→36fps), and high-end devices (such as A16 chip) can stably run 4K / 60fps high-definition models.

[0055] The actual measurement data shows that the memory occupation is reduced by 57% (from 2.1GB→0.9GB), which significantly prolongs the battery life of mobile devices.

[0056] 2. Network disaster recovery capability enhancement

[0057] Multi-level cache architecture:

[0058] The edge node deployment reduces the core area model loading delay by 76% (380ms→92ms), and still maintains 41fps smooth operation in weak network environment (2Mbps).

[0059] When the network is disconnected, the local cache provides LOD6 level models within a radius of at least 50 meters, and the service availability reaches 92% (traditional scheme is 0%).

[0060] 3. Improved visual experience

[0061] Light consistency processing:

[0062] The environment light probe real-time sampling technology reduces the virtual object shadow direction error to <3°, and the material reflection coefficient matching accuracy is improved to 89%.

[0063] Mixed rendering strategy (Light tracing + Shadow Map) makes the shadow quality PSNR value > 38dB while keeping the frame rate fluctuation < 5%.

[0064] 4. Scene expansion capability

[0065] Cross-scene adaptation:

[0066] The library scene realizes centimeter-level positioning (UWB + SLAM fusion algorithm, error ± 10cm), and supports real-time loading and interaction of 5000 book models.

[0067] In the multi-user collaborative mode of the science museum, the 8-user concurrent operation delay is < 80ms, and the model synchronization accuracy is 0.1mm level.

[0068] 5. Reduction of operating costs

[0069] Bandwidth saving:

[0070] Draco compression algorithm reduces single-user traffic consumption by 79% (from 210MB / 10min to 42MB / 10min), and saves bandwidth cost of more than 3 million yuan under the scale of ten thousand users per year.

[0071] Edge node difference synchronization technology reduces 90% of data transmission (daily synchronization volume from 800MB to 80MB). BRIEF DESCRIPTION OF DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained according to the structures shown in these drawings without creative labor for those skilled in the art.

[0073] Figure 1 is the scene AR guide system architecture diagram supporting 3D model dynamic loading in the embodiment of the present application;

[0074] Figure 2 is the scene AR guide data processing flowchart supporting 3D model dynamic loading in the embodiment of the present application.

[0075] The implementation of the present application, functional characteristics and advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0076] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0077] The specific implementation completely shows the landing details of the technical solutions by using real scene data, reproducible algorithm parameters and comparative experiment data, and refers to Figure 1 and Figure 2 :

[0078] Embodiment 1: System deployment and running process (taking the Palace Museum application as an example)

[0079] 1. Data acquisition and preprocessing

[0080] Laser radar scanning:

[0081] RIEGL VZ-400i is used to scan the Taihe Palace area, the scanning accuracy is set to 2mm@10m, and point cloud data (about 1.2 billion points of single building point cloud) is obtained

[0082] The eaves, dougong and other complex structures are focused on supplement scanning to ensure that the point cloud density of the key parts is greater than or equal to 500 points / ㎡

[0083] Unmanned aerial vehicle aerial photography:

[0084] DJI Matrice 300 RTK is equipped with Zenmuse P1 camera, the flight height is 30m, the heading overlap rate is 80%, and the side overlap rate is 70%;

[0085] 0.5cm resolution orthographic images are generated, and 4096x4096 texture maps are generated by Photoscan.

[0086] Model fusion and optimization:

[0087] In Blender:

[0088] bpy.ops.import_mesh.ply(filepath="Taihe Palace point cloud.ply")

[0089] bpy.ops.texture.mapping_project(scale=(1,1,1))#projected map

[0090] bpy.ops.export_scene.gltf(filepath="TaiheDian.gltf", export_format='GLTF_SEPARATE')

[0091] Output model contains:

[0092] Vertex count: 2,134,578

[0093] Triangle count: 4,231,995

[0094] Texture size: 4K PBR material (metallic / roughness dual-channel map)

[0095] 2. Dynamic loading strategy implementation

[0096] Device performance adaptation (using Huawei P50 Pro as an example):

[0097] Mali-G78MP22 GPU (2.4 TFLOPS) detected, according to the formula:

[0098]

[0099] Select LOD1 level model (triangle count ≈ 1,057,998)

[0100] Network status evaluation:

[0101] Actual 4G network bandwidth 8Mbps, triggering loading strategy:

[0102] Basic model: LOD3 level (triangle count ≈ 264,499)

[0103] Key model (user gaze area): LOD1 level

[0104] Preloading mechanism:

[0105] Tourists move at a walking speed of 1.2m / s, preloading radius calculation: R = max(50, (1.2x3.6)x10 / 3.6) = 12 meters → automatically expand to the minimum 50-meter preloading Taihe Gate, Zhonghe Hall and other buildings in front of 50 meters.

[0106] 3. Actual performance data

[0107]

[0108] Example 2: Data processing and rendering optimization (Huangshan scenic area scene as an example) 1. LOD grading optimization process

[0109] Model simplification parameters:

[0110] Grading the Welcoming Pine model using Simplygon 4.0:

[0111]

[0112] The topology changes after generating nine levels of LOD:

[0113]

[0114] 2. Implementation of hybrid rendering technology

[0115] Lighting consistency processing:

[0116] AR Foundation environment probes capture lighting parameters in real time:

[0117] LightEstimation lightEstimation=arCamera.GetComponent <lightestimation>float ambientIntensity = lightEstimation.ambientIntensity.Value;

[0118] Color colorCorrection = lightEstimation.colorCorrection.Value;

[0119] Dynamic adjustment of virtual model Shader parameters:

[0120]

[0121] Shadow hierarchical rendering:

[0122] Rock model within 10m distance: Ray-tracing soft shadows enabled (2.3ms per frame)

[0123] Mountain model beyond 10m distance: Cascade Level = 4, resolution 512x512

[0124] 3、Weak network environment solution

[0125] Network bandwidth drops to 1.8Mbps:

[0126] (1) Trigger wireframe mode switching:

[0127] Patch number from 291,600 (LOD3) to 58,320 (wireframe mode)

[0128] Texture compression to 512x512 resolution

[0129] (2) Start UDP emergency transmission channel:

[0130] Keyframe data transmission priority (head pose matrix, visitor positioning coordinates)

[0131] Transmission interval from 100ms to 300ms

[0132] (3) Local cache automatic expansion:

[0133] From the default 512MB to 1GB, cache radius extended to 100 meters

[0134] Comparison of actual measurement results:

[0135]

[0136] Example 3: Edge node deployment scheme (Hangzhou West Lake scenic area as an example)

[0137] 1、Edge computing node configuration

[0138] Hardware deployment:

[0139] 1 Huawei Atlas 500 (configuration: Ascend 310 chip, 8GB RAM) per square kilometer;

[0140] Cover key scenic spots such as Broken Bridge and Leifeng Pagoda, forming a honeycomb service network.

[0141] Data synchronization mechanism:

[0142] Incremental synchronization from 2:00 to 4:00 every day:

[0143] def sync_diff(old_hash, new_hash):

[0144] changed_blocks = compute_xor(old_hash, new_hash)

[0145] return zlib.compress(changed_blocks) # compression rate 85%

[0146] Single node daily synchronization data volume about 120MB (original data volume 800MB)

[0147] 2, Localized service process

[0148] Request response logic:

[0149] 1. User equipment preferentially initiates request to the nearest edge node (response time ≤ 50ms);

[0150] 2. Node checks local model version:

[0151] Version consistent: directly return LOD3 level model

[0152] Version expired: pull difference block from cloud (difference rate < 15%)

[0153] 3. Emergency mode is enabled when offline: provide LOD9 level simplified model + black and white texture

[0154] Deployment effect:

[0155] Average loading delay is reduced from 380ms (pure cloud) to 92ms

[0156] Core area network traffic is reduced by 72%

[0157] Technical effect verification case

[0158] Test scenario: Beijing Summer Palace Seventeen-hole Bridge area

[0159] Test equipment:

[0160] High-end machine: iPhone 14Pro (A16 chip, 5.6TFLOPS)

[0161] Low-end machine: Redmi Note 11 (Snapdragon 680, 0.8TFLOPS)

[0162] Comparison data:

[0163]

[0164]

[0165] Example 4: Large-scale library AR tour application (taking the East Library of Shanghai Library as an example)

[0166] Environmental characteristics:

[0167] Indoor multi-story building structure (5-story stereoscopic book library);

[0168] High-density bookshelf arrangement (spacing 1.5m);

[0169] Need to display digital content such as 3D models of books and holographic images of authors.

[0170] Technical challenges:

[0171] Indoor GPS failure, need centimeter-level positioning;

[0172] Large number of fine models real-time rendering requirements (single book model polygon number ≥50,000); Network congestion caused by dense flow.

[0173] 2. Specific technical implementation

[0174] (1) Data collection and modeling

[0175] Bookshelf scanning:

[0176] Faro Focus S 350 laser scanner is used, with scanning accuracy ±1mm, generating bookshelf point cloud model (single layer data volume about 800 million points);

[0177] Artec Eva handheld scanner is used for precious ancient books to obtain 0.1mm precision texture model.

[0178] Digital content production:

[0179] Author holographic image: 8K resolution dynamic model (60fps, 2 million polygons) is generated through volume capture technology;

[0180] Book 3D models: PBR materials created using Substance Painter (4K normal map + 2K roughness map).

[0181] (2) Indoor positioning enhancement

[0182] Multi-source fusion positioning system:

[0183]

[0184] Positioning data fusion algorithm:

[0185] def fusion_position(uwb, slam, bluetooth):

[0186] weight = [0.6, 0.3, 0.1] # confidence weights

[0187] return (uwb * weight[0] + slam * weight[1] + bluetooth * weight[2]) / sum(weights)

[0188] (3) Dynamic loading strategy optimization

[0189] Shelf area hierarchical loading:

[0190]

[0191] Focus book preloading:

[0192] Preload related book models according to user borrowing records (historical data);

[0193] When a user gazes at a certain type of book label for more than 3 seconds, trigger the preloading of similar book models.

[0194] (4) Edge computing node deployment

[0195] Hardware configuration:

[0196] Deploy 2 NVIDIA Jetson AGX Orin (32GB RAM, 275TOPS) per floor;

[0197] Storage capacity: 1TB NVMe SSD (store all LOD3 level models of this floor).

[0198] Performance test data

[0199]

[0200] Example 5: Science Museum interactive exhibition application (take the Space Exhibition Area of China Science and Technology Museum as an example)

[0201] 1. Technical solution highlights

[0202] High-precision spacecraft model interaction:

[0203] Long March 5 rocket model: LOD0 level patch number 520 million (including internal structure details);

[0204] Dynamic separation demonstration: hierarchical loading of fairing, booster and other detachable component models.

[0205] Multi-person collaborative AR experience:

[0206] Support 8 people operating the same spacecraft model at the same time;

[0207] Distributed rendering technology is adopted, with the main device rendering the core model and the edge nodes handling auxiliary components.2. Key technology implementation

[0208] (1) Dynamic model splitting and loading

[0209] Componentized model management:

[0210]

[0211] Interactive trigger loading:

[0212] When the user's finger is within 0.5m of the booster, load the LOD2 level detachable model (patch number 120,000); when the disassembly animation is playing, automatically downgrade the unfocused components to LOD5 level.

[0213] (2) Adaptive lighting rendering

[0214] Exhibition hall light synchronization:

[0215] Real-time acquisition of exhibition area light parameters (color temperature, intensity) through DMX512 protocol;

[0216] Virtual model lighting dynamic adjustment algorithm:

[0217]

[0218] Reflection environment construction:

[0219] Update the environment cubemap every 30 minutes (Cubemap Resolution: 1024x1024).

[0220] 3. Implementation effect comparison

[0221]

[0222] Technical solution universality

[0223] 1. Cross-scene parameter configuration table:

[0224]

[0225] Core algorithm reuse:

[0226] Dynamic loading decision tree can be used across scenarios (adjust distance threshold and LOD mapping relationship);

[0227] Multi-level cache mechanism only needs to modify the storage strategy in different scenarios (library focuses on document model caching, science museum focuses on interactive component caching).

[0228] Special scene expansion:

[0229] Ancient book restoration AR assistance: Implement LOD12 level ultra-high precision model (facet number> 2000 million) for special collection documents, use WLAN 6E special channel transmission during loading;

[0230] Space capsule virtual experience: When detecting VR headset connection, automatically enable LOD0 level model + 8K texture, frame rate locked at 90fps.

[0231] The technical scheme of the present application solves the key technical problems of poor device adaptability, high network dependence and serious visual distortion in the field of AR guide through the innovative design of dynamic resource scheduling algorithm, multi-level cache architecture and hybrid rendering pipeline. Practical application data shows that in typical scenes such as scenic spots, libraries and science museums, the system performance indicators are superior to existing solutions, providing reliable technical support for large-scale commercialization.

[0232] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation, direct / indirect application in other related technical fields within the inventive concept of the present application, and the contents of the present application specification and drawings are included in the patent protection scope of the present application.< / lightestimation>

Claims

1. The scenic area AR guide system that supports dynamic loading of 3D models is characterized by: include: Data acquisition module: Use laser radar to scan the terrain of the scenic area, with a point cloud density of ≥200 points / ㎡; Obtain 0.5cm resolution texture maps through drone aerial photography; Use the fusion process of point cloud and texture to generate glTF 2.0 format 3D model including normal map and AO map; Dynamic loading control modules, including: Device performance detection unit: obtains the GPU model through the OpenGL ES 3.2 interface and queries the preset database to match the computing power level; Network status evaluation unit: uses weighted moving average algorithm to calculate real-time bandwidth. The formula is: When the bandwidth falls below the threshold of 2Mbps, the downgrade loading strategy is triggered; Position sensing unit: combines GPS and visual SLAM to achieve sub-meter positioning accuracy; Multi-level cache architecture, including: Local storage layer: caches all LOD0 models within a 50m radius of the user; Edge node layer: Huawei Atlas 500 intelligent edge devices are deployed to store the full LOD3 model of the scenic area, with a response time of less than 100ms. Cloud storage layer: Alibaba Cloud OSS stores the original LOD0 model and supports block transmission via HTTPS protocol; AR rendering engine: Use Unity 2022LTS version and integrate AR Foundation 5.0 framework; Lighting consistency processing: The scene HDR lighting information is collected in real time through the ambient light probe, and the diffuse reflection coefficient of the virtual model is adjusted synchronously.

2. The scenic area AR guide system supporting dynamic loading of 3D models according to claim 1 is characterized in that: The network status evaluation unit includes a packet loss rate compensation mechanism, which automatically switches to the UDP protocol to transmit key frame data when it detects that the packet loss rate is greater than 5%.

3. The scenic area AR navigation system supporting dynamic loading of 3D models according to claim 1 is characterized in that: The edge node layer adopts a differential synchronization algorithm and synchronizes data with the cloud through a JSON differential file (difference rate <15%) from 23:00 to 5:00 every day, and the synchronization bandwidth occupancy rate is ≤20%.

4. A method for processing scenic area AR guide data that supports dynamic loading of 3D models, characterized in that: The following steps are involved: S1. Model preprocessing Perform nine-level LOD optimization on the original 3D model: LOD0 retains 100% of the number of faces; The number of LODn level faces decreases according to the formula: N n =N n-1 ×(0.65-0.05n)(1≤n≤9) S2. Dynamic loading decision Device adaptation strategy: Select the LOD level based on GPU computing power, matching rules: Network adaptation strategy: Bandwidth classification and corresponding loading plan: Step S3: Preloading and cache updating Preload radius calculation based on user movement speed: The LRU algorithm is used to manage the local cache, and the 20% oldest models are automatically deleted when the storage capacity exceeds 512MB.

5. The scenic area AR guide data processing method supporting dynamic loading of 3D models according to claim 4 is characterized in that: In step S2, a priority promotion strategy is implemented for the gaze area model. The user's gaze coordinates are identified through the eye tracking camera, and the model within the 10° cone area of ​​the line of sight center is forced to load the LOD0 level.

Images