Exhibition hall three-dimensional modeling intelligent optimization system based on multi-modal data fusion

Abstract

The invention relates to the technical field of computer vision and three-dimensional reconstruction, and discloses an exhibition hall three-dimensional modeling intelligent optimization system based on multi-modal data fusion, and the system comprises a data collection module which is configured to synchronously obtain laser radar point cloud data, a multispectral image sequence and inertial measurement unit data; the preprocessing module is used for receiving the output of the data acquisition module, aligning a multi-source sensor coordinate system through a space-time calibration algorithm, and separating a static scene from a dynamic interference element by using a dynamic segmentation network; and the multi-modal fusion module is used for receiving the preprocessed data and carrying out adaptive weighted fusion on the geometric features of the laser radar and the visual texture features through a cross-modal attention mechanism. According to the invention, through multi-modal data fusion and a dynamic scene adaptive mechanism, the modeling precision and the real-time updating capability in a complex exhibition hall environment are significantly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and three-dimensional reconstruction, and in particular to an intelligent optimization system for three-dimensional modeling of exhibition halls based on multimodal data fusion. Background Art

[0002] In the field of digital exhibition hall modeling, traditional 3D reconstruction technology primarily relies on a single sensor data source and static scene assumptions, making it difficult to meet the high-precision modeling requirements in complex dynamic environments. While existing laser scanning-based methods can achieve millimeter-level geometric accuracy, they are limited by the fixed viewing angle of the device and scanning efficiency, making it difficult to fully capture the details of large-scale exhibition hall spaces and prone to data loss on reflective surfaces. While visual SLAM solutions offer real-time advantages, they are affected by lighting variations and texture loss, and the integrity of geometric reconstruction is heavily dependent on environmental conditions, making it difficult to guarantee the modeling quality of highly reflective surfaces such as exhibition hall artwork.

[0003] While current neural radiance field technology shows potential for modeling static objects, its global continuous representation leads to inefficient training for large-scale scenes and a lack of ability to handle dynamic elements. Existing methods rely on full scene retraining to accommodate layout changes, resulting in wasted computing resources and insufficient real-time performance. Furthermore, traditional 3D models lack semantic information integration, making them difficult to support the intelligent interaction requirements of digital exhibition halls. Existing semantic segmentation methods often rely on post-processing labeling, which leads to spatial alignment errors between semantic and geometric features.

[0004] Furthermore, at the engineering application level, existing systems often utilize centralized computing architectures, resulting in rendering delays and bandwidth pressure, making it difficult to meet the real-time interaction requirements of mobile devices. Regarding data management, traditional solutions lack version control and differential update mechanisms, resulting in high costs for historical status tracing and dynamic scene maintenance. These technical bottlenecks severely restrict the usability and scalability of digital exhibition hall systems in practical applications. Summary of the Invention

[0005] The purpose of the present invention is to provide an intelligent optimization system for exhibition hall three-dimensional modeling based on multimodal data fusion, which solves the problems of insufficient multi-source data fusion, poor adaptability to dynamic scenes, lack of semantic information and low efficiency of large-scale scene rendering in the existing exhibition hall three-dimensional modeling technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: The exhibition hall 3D modeling intelligent optimization system based on multimodal data fusion includes: a data acquisition module configured to simultaneously acquire lidar point cloud data, multispectral image sequences, and inertial measurement unit data; a preprocessing module that receives the output of the data acquisition module, aligns the multi-source sensor coordinate systems using a spatiotemporal calibration algorithm, and separates static scenes from dynamic interference elements using a dynamic segmentation network; The multimodal fusion module receives preprocessed data and adaptively weights the lidar geometric features and visual texture features through a cross-modal attention mechanism; The neural radiation field modeling module constructs a block-based neural radiation field based on the fused features and maintains the geometric continuity between blocks through interface consistency constraints; The reflection-radiation coupling module injects material reflection property parameters into the radiation field rendering pipeline and establishes bidirectional differential constraints to optimize surface material performance; Dynamic update module, triggers incremental training based on scene change detection results and updates neural radiation field parameters; The semantic fusion module injects semantic information in stages during the geometric modeling process to generate a semantically queryable 3D model; A lightweight output module converts the optimized radiation field model into a compact representation that supports real-time rendering and outputs it to the visualization terminal through a standardized interface.

[0007] Preferably, the data acquisition module includes: The LiDAR collects millimeter-level precision point cloud data at a frequency of 10 Hz; The multispectral camera synchronously acquires RGB-D image streams with a resolution of no less than 1920×1080; The inertial measurement unit records six-degree-of-freedom posture data at a frequency of 90Hz-120Hz.

[0008] Preferably, the spatiotemporal calibration algorithm of the preprocessing module solves the nine-degree-of-freedom transformation matrix by minimizing the reprojection error: T=argmin∑||π(Rp i +t)-u i || 2 ; Where R is the rotation matrix, t is the translation vector, π(·) is the camera projection model, and p is the ray sampling probability distribution function.

[0009] Preferably, the dynamic segmentation network adopts a dual-stream architecture: The static branch uses 3D sparse convolution to process point cloud data, and the dynamic branch extracts motion features through the optical flow network, and finally generates a dynamic mask through a gating mechanism.

[0010] Preferably, the cross-modal attention mechanism of the multimodal fusion module satisfies: Among them, the query vector Q comes from the lidar feature, and the key-value pairs K and V are generated by splicing visual features.

[0011] Preferably, the block strategy of the neural radiation field modeling module divides the scene into N×N independent modeling blocks and imposes interface constraints: in, is a set of adjacent block pairs, x ij The interface sampling point.

[0012] Preferably, the differential constraints established by the reflection-radiation coupling module include: Among them, f r is the bidirectional reflectance distribution function based on the microsurface model.

[0013] Preferably, the incremental training of the dynamic update module adopts an adaptive learning rate strategy: η t =η0·exp(-α·t); Among them, α is the decay coefficient and t is the number of update iterations.

[0014] Preferably, the phased injection of the semantic fusion module includes: The geometric embedding stage encodes semantic labels into mesh vertex features; The sampling stage adjusts the ray distribution through attention weights; A pixel-level semantic consistency loss is imposed during the rendering phase.

[0015] Preferably, the lightweight output module is: Edge computing nodes perform feature extraction and data compression; The cloud server performs radiation field calculations and parameter updates; The WebGL engine enables real-time rendering on the browser side.

[0016] In summary, the present invention includes at least one of the following beneficial technical effects: 1. This invention effectively overcomes the limitations of a single sensor by integrating cross-modal features from lidar, vision, and inertial data, achieving high-precision alignment of geometry and texture in complex lighting and reflective material scenarios. Its innovative cross-modal attention mechanism adaptively weights the contributions of different data sources, addressing the feature mismatch problem of traditional methods in exhibition hall environments.

[0017] 2. The block-based neural radiation field architecture of the present invention is combined with an incremental update mechanism to support real-time modeling of exhibition hall layout changes and exhibit replacements. The dynamic consistency constraint and motion compensation algorithm effectively distinguish between real scene changes and acquisition equipment movement, ensuring the accuracy and stability of model updates.

[0018] 3. The three-level progressive semantic injection pipeline of the present invention embeds knowledge graph information into the modeling process to generate a three-dimensional model with interpretable semantic attributes. The semantic association modeling based on the graph attention network lays the foundation for providing advanced applications such as intelligent navigation and exhibit retrieval in digital exhibition halls.

[0019] 4. The edge-cloud collaborative architecture and vector quantization compression technology of the present invention reduce transmission bandwidth by more than 90% while maintaining visual fidelity. Layered detail control and differential storage strategies enable efficient management of large-scale scene data and support multi-terminal real-time interactive applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a system architecture diagram of the present invention. DETAILED DESCRIPTION

[0021] The following is combined with Figure 1 , the present invention is described in further detail.

[0022] The present invention provides an exhibition hall three-dimensional modeling intelligent optimization system based on multimodal data fusion. include: The data acquisition module is configured to simultaneously acquire lidar point cloud data, multispectral image sequences, and inertial measurement unit data. In this embodiment, the data acquisition module utilizes a multi-sensor collaborative acquisition strategy, specifically comprising a combination of a lidar unit, a multispectral vision unit, and an inertial measurement unit. The lidar unit preferably utilizes a solid-state rotary scanning structure, emitting laser pulses and receiving reflected signals to acquire millimeter-level precision three-dimensional point cloud data of the scene. Its scanning plane covers an angular range of -30° to +45° vertically, ensuring complete geometric capture of exhibits at varying heights within the exhibition space.

[0023] The multispectral vision unit integrates a visible light imaging module and a depth perception module. The visible light imaging module is equipped with a high dynamic range sensor to cope with the complex lighting conditions of the exhibition hall. The depth perception module preferably uses a structured light projection solution to obtain real-time scene depth information through deformation analysis of the coded spot pattern. Spatial registration of the vision unit and the lidar unit is achieved through a pre-calibrated extrinsic parameter matrix to ensure the consistency of multi-source data in the spatial coordinate system.

[0024] In this embodiment, the inertial measurement unit (IMU) incorporates a three-axis gyroscope and accelerometer to continuously collect the device's six-degree-of-freedom motion parameters. This is aligned with the LiDAR pulse signal via a hardware synchronization interface, establishing a unified time base system. The spatiotemporal synchronization process specifically involves triggering the camera exposure when the LiDAR completes a single scan, while simultaneously recording the IMU's interpolated pose data at that moment, forming a multimodal data package with temporal and spatial correlation.

[0025] The calibration method for the data acquisition module includes two stages: offline joint calibration and online dynamic correction. In the offline calibration stage, a specially designed composite calibration target is arranged, and the target surface is set with a combination of high reflectivity markers and a checkerboard pattern. The spatial transformation relationship between sensors is solved using a reprojection error minimization algorithm: Among them, π(·) represents the camera projection model, is the laser radar point cloud coordinate, is the corresponding image pixel coordinate, is the inertial measurement unit data, is the ground truth motion parameter, and λ is the weight coefficient.

[0026] During the online dynamic correction phase, feature point tracking and inertial measurement data are tightly coupled and optimized to compensate for calibration parameter drift caused by device vibration or temperature changes in real time. Specifically, during continuous acquisition, stable feature points are extracted from the scene, cross-modal feature associations are established, and calibration parameters are dynamically updated using an extended Kalman filter.

[0027] In this embodiment, the data acquisition module is equipped with an adaptive scanning strategy generation unit, which analyzes scene complexity in real time and dynamically adjusts sensor operating parameters. In areas with densely populated exhibits, the lidar scanning frequency and camera sampling rate are automatically increased; in open areas, the acquisition density is reduced to optimize resource consumption. Scanning path planning utilizes a reinforcement learning-based decision model, which evaluates metrics such as scene coverage and feature richness to generate the optimal device trajectory.

[0028] The multispectral vision unit's operating mode adaptively switches based on the exhibition hall's lighting conditions, automatically activating the fill light module and extending exposure time in low-light environments, and activating multi-frame synthesis technology in high-light-ratio scenes. The depth perception module automatically switches to time-of-flight measurement mode when detecting transparent or highly reflective surfaces, improving depth measurement accuracy by calculating the phase difference of laser pulses.

[0029] In this embodiment, the data acquisition module incorporates an abnormal data detection and repair mechanism. By establishing a cross-validation model for point cloud, image, and inertial data, it identifies and eliminates invalid measurement data caused by factors such as human movement and temporary occlusion in real time. If missing data is detected in key exhibit areas, a local rescan process is automatically triggered to ensure the integrity and reliability of the collected data.

[0030] The preprocessing module receives the output of the data acquisition module, aligns the multi-source sensor coordinate system through a spatiotemporal calibration algorithm, and uses a dynamic segmentation network to separate static scenes from dynamic interference elements; In this embodiment, the preprocessing module achieves spatiotemporal alignment and dynamic interference suppression for multi-source heterogeneous data through a multi-stage data processing process. Specifically, the preprocessing module receives raw laser point clouds, multispectral images, and inertial measurement data from the data acquisition module and performs spatiotemporal sensor calibration, dynamic element segmentation, and four-dimensional voxelization operations, providing standardized input for subsequent multimodal fusion.

[0031] The spatiotemporal calibration process uses a hierarchical optimization strategy. First, based on the initial extrinsic parameter matrix obtained during the offline calibration phase, the lidar point cloud is projected into the camera coordinate system to form a virtual depth map. By maximizing the structural similarity between the virtual depth map and the measured depth map, the rotation and translation parameters are iteratively optimized: Among them, D proj Denotes the projected depth map, D real is the depth map measured by the camera, and SSIM is the structural similarity metric function. Kinematic constraints of inertial measurement data are further introduced to establish a tightly coupled optimization model to eliminate cumulative errors.

[0032] In this embodiment, dynamic element segmentation is achieved through a two-stream neural network architecture. The static processing branch uses a sparse convolutional network to extract point cloud geometric features and construct a voxel representation of the scene; the dynamic processing branch uses an optical flow network to analyze the motion patterns between consecutive image frames and combines it with a semantic segmentation network to identify moving objects. The features of the two branches are fused through a gated attention mechanism: M d =σ(W s ·F s +W d ·F d ); Among them, F s is the geometric feature tensor output by the static branch, F d is the motion feature extracted by the dynamic branch, σ represents the Sigmoid activation function, W s and W d is a learnable weight matrix. The generated dynamic mask M d Used to filter out temporary interference elements such as visitors.

[0033] The four-dimensional voxelization process incorporates the time dimension into the spatial encoding system to construct a spatiotemporal continuous scene representation. Specifically, the spatiotemporal discretization operation is performed on the point cloud data: Among them, φ i (·) is a three-dimensional space basis function, preferably represented by a truncated signed distance function (TSDF); ψ i (·) is a temporal basis function that uses an exponential decay model to characterize the timeliness weight of the observation data. This representation method effectively integrates data collected over multiple time periods, enhancing the integrity of scene reconstruction.

[0034] In this embodiment, the preprocessing module also includes an abnormal data repair unit, which identifies sensor noise and acquisition anomalies by establishing a cross-modal consistency check model. Outliers in the LiDAR point cloud are repaired by interpolation using the spatial continuity characteristics between adjacent frames. For overexposed areas in the image sequence, texture details are reconstructed by combining multispectral information with point cloud reflectance intensity data.

[0035] The spatiotemporal calibration process further includes an online adaptive correction mechanism that dynamically adjusts the extrinsic parameter matrix by continuously tracking the reprojection error of stable scene feature points. When the calibration parameter drift exceeds the preset threshold, the local optimization process is automatically triggered: Where ΔT represents the calibration parameter correction, p k and u k This mechanism effectively compensates for calibration errors caused by device vibration and temperature drift, ensuring the long-term alignment accuracy of multi-source data.

[0036] Through the above technical solution, the preprocessing module achieves high-precision spatiotemporal alignment and dynamic interference suppression of multi-sensor data, providing standardized and denoised multimodal input for subsequent modeling processes, ensuring the robustness of the system in complex exhibition hall environments.

[0037] The multimodal fusion module receives preprocessed data and adaptively weights the lidar geometric features and visual texture features through a cross-modal attention mechanism; In this embodiment, the multimodal fusion module achieves efficient feature fusion of lidar, visual, and inertial data through a cross-modal attention mechanism and uncertainty weighting strategy. This module receives preprocessed spatiotemporally aligned point cloud data, multispectral imagery, and inertial measurement information, and generates a unified scene representation through hierarchical feature extraction and adaptive fusion, providing multi-source complementary feature input for subsequent neural radiation field modeling.

[0038] The feature encoding stage adopts a divide-and-conquer strategy for heterogeneous data. For lidar point cloud data, multi-scale geometric features are extracted through a sparse convolutional network: in, Represents the first layer of point cloud features, is the three-dimensional sparse convolution kernel parameter, and the output features retain the spatial sparse characteristics of the point cloud, reducing the computational complexity.

[0039] A dual-branch encoding architecture is used for visual data, processing RGB images and depth maps separately: F RGB =ResNet(I),F Depth =DepthNet(D); Among them, DepthNet adopts an encoder-decoder structure and retains multi-scale depth features through skip connections.

[0040] Cross-modal attention fusion constructs a geometry-guided texture fusion mechanism, defining the query vector from the lidar feature and the key-value pair from the visual feature: Q=W Q F LiDAR ,K=W K [F RGB ; F Depth ],V=W V [F RGB ; F Depth ]; Among them, W Q : Query the projection matrix and map the lidar feature dimension C to C q W K ,W V : Key / value projection matrix, mapping the concatenated visual feature dimension 384 to C k .

[0041] Attention weights are calculated by scaling the dot product: Where: Q is the query matrix (geometric features), K, V is the key / value matrix (visual features), d = C k is a scaling factor to prevent the gradient explosion of the dot product result, and softmax is a row normalized exponential function to generate the attention weight matrix.

[0042] This mechanism enables geometric features to dynamically focus on relevant texture information, solving the texture-geometry mismatch problem caused by reflective surfaces. The uncertainty weighted model dynamically adjusts the fusion weight by evaluating the confidence of each modal data and defines the uncertainty of lidar ranging: Where: r is the measurement distance, α and β are calibration parameters.

[0043] Visual depth uncertainty is predicted by the network: σ Vis =f Uncertainty (I,D); Among them, f Uncertainty is the depth uncertainty prediction network, is the standard deviation map with the same dimensions as the output depth map, is the IRGB image input, and is the original depth map input.

[0044] The final fusion weight is calculated as: Among them, σ i The measurement uncertainty of the i-th mode (σ LiDAR or σ Vis ), the normalized modal weight satisfies ∑w i =1.

[0045] This strategy automatically enhances the contribution of visual data in areas where lidar data is unreliable (such as transparent glass display cases), improving system robustness.

[0046] The motion compensation unit corrects the spatiotemporal offset through inertial data and converts the IMU pose change into an affine transformation in feature space: F ′ =F·W T ΔT; Where: ΔT is the pose transformation matrix obtained by IMU integration, W T is the learnable feature space projection parameter, F is the original feature matrix, F ′ is the feature matrix after motion compensation.

[0047] This operation eliminates feature misalignment caused by device motion and ensures fusion consistency in dynamic scenes.

[0048] The neural radiation field modeling module constructs a block-based neural radiation field based on the fused features and maintains the geometric continuity between blocks through interface consistency constraints; In this embodiment, the Neural Radiance Field Modeling Module utilizes a Block-Neural Radiance Field (Block-NeRF) architecture, combining dynamic consistency constraints with a physical reflectance model to achieve high-fidelity 3D reconstruction of large-scale exhibition hall scenes. This module receives the cross-modal feature tensor output by the Multimodal Fusion Module and generates a differentiable scene representation through spatial partitioning, radiation field modeling, and global optimization, providing the foundation for subsequent dynamic updates and lightweight output.

[0049] The block modeling strategy divides the exhibition hall scene into several independent modeling blocks, and the block size is adaptively adjusted according to the complexity of the scene. Preferably, the benchmark division size is used for the general exhibition area, and the high-precision subdivision mode is enabled for the key exhibit areas. Each block maintains an independent neural radiation field model: Where: γ(·) is the position encoding function, which maps the three-dimensional coordinate x and the observation direction d to the high-dimensional feature space, F fuse is the cross-modal feature from the multimodal fusion module, σ i is the volume density of spatial points in block i, c i is the radiation color associated with the viewing direction, θ i It is a block-specific network parameter.

[0050] The consistency constraint between blocks is realized by the interface loss function, which uniformly samples the point set {x ij}, to enforce consistency between color and density predictions: in: is a set of adjacent block pairs, λ is the weight coefficient for balancing color and density consistency, x ij is the spatial sampling point at the interface.

[0051] Physical reflection modeling is achieved by coupling the bidirectional reflectance distribution function (BRDF) with the neural radiation field. Define the microsurface reflection model: Where: D(h) is the normal distribution function, which describes the statistical distribution of the microsurface normal h, G(ω i ,ω o ) is the geometric attenuation term, which describes the microsurface shielding effect, F(ω i ) is the Fresnel reflection term, and n is the macroscopic surface normal.

[0052] Establishing a gradient propagation path between the radiation field prediction and the physical reflection model through a differentiable rendering pipeline: This differential relationship enables the network to jointly optimize geometric surface properties and material reflectance parameters from multi-view observation data.

[0053] Dynamic element processing is achieved through a spatiotemporal encoding mechanism, which introduces motion consistency constraints based on the four-dimensional spatiotemporal coordinates (x, t): Where: σ(x,t) is the volume density of the space-time coordinate point.

[0054] The constraint term suppresses the time-varying noise in the static region while preserving the true dynamic changes. For detected persistent dynamic objects, an independent sub-radiation field modeling process is started to avoid contaminating the main scene model.

[0055] Adaptive sampling optimization improves reconstruction efficiency through importance sampling strategy and defines sampling probability distribution: in: is the predicted density on ray r, is the density spatial gradient, β is the gradient sensitivity coefficient.

[0056] This distribution guides the sampling points to focus on high-density areas and geometric edges, reducing invalid sampling while ensuring reconstruction accuracy.

[0057] The reflection-radiation coupling module injects material reflection property parameters into the radiation field rendering pipeline and establishes bidirectional differential constraints to optimize surface material performance; In this embodiment, the reflection-radiation coupling module achieves high-fidelity rendering of complex materials in exhibition hall scenes by deeply integrating physical reflection models with neural radiation fields. This module injects material reflectance property parameters into the radiation field rendering pipeline and establishes a physically based differentially constrained optimization mechanism, effectively addressing the rendering distortion issues of traditional neural radiation fields on reflective and transparent surfaces.

[0058] The bidirectional reflectance model integration uses the microsurface theory framework to embed the bidirectional reflectance distribution function (BRDF) into the radiation field rendering equation. Define the differentiable rendering formula: L o (x,ω o )=∫ Ω f r (ω i ,ω o )L i (x,ω i )(n·ω i )dω i +L vol (x,ω o ); Where: L o is the outgoing radiation brightness, f r is the bidirectional reflectance distribution function, L i is the incident radiation brightness, n is the surface normal, L vol is the volume scattering contribution.

[0059] Unify surface reflection and volume rendering in a differentiable framework to achieve end-to-end optimization of material optical properties.

[0060] The material parameter injection mechanism is implemented through feature space mapping to build a material attribute encoder: m=g φ (F fuse ,z mat ); Among them: F fuse is the multimodal fusion feature, z mat Embed vector for material type, g φ is a multi-layer perceptron encoding network, and m is the output reflection parameter set (roughness, metallicity, etc.).

[0061] The encoded material parameters dynamically adjust the normal distribution function D and Fresnel term F in the BRDF model, so that the radiation field rendering can adapt to different material characteristics.

[0062] Differential constrained optimization is achieved by establishing the gradient propagation path of the surface normal and the radiation field. Define the normal-radiation coupling loss: This constraint forces the normal changes predicted by the neural network to conform to the laws of physical reflection, enhancing the realism of the material performance.

[0063] The dynamic material adaptation module automatically loads the pre-trained parameter basis set for the detected new material type through the online material library retrieval mechanism. Preferably, a meta-learning framework is used to build the parameter generator: Where: h ψ For the hypernetwork, embed the vector according to the new material, To generate adaptive network parameters and achieve fast adaptation with few samples.

[0064] Dynamic update module, triggers incremental training based on scene change detection results and updates neural radiation field parameters; In this embodiment, the dynamic update module continuously optimizes the exhibition hall's 3D model and adapts to changes in the scene in real time through incremental training and selective parameter updates. This module receives scene state analysis data from the semantic fusion module and, combined with multimodal sensory input, dynamically adjusts the neural radiation field parameters to maintain consistency between the model and the physical scene.

[0065] The change detection unit calculates the difference between the continuous model rendering results and the real-time observation data based on multi-scale feature difference analysis. Define the structural similarity loss and depth consistency loss: Among them, I render and D render They are the RGB image and depth map rendered by the current model, I obs and D obsis the real-time observation data, and a is the balance coefficient. The model update process is triggered when the preset threshold is exceeded.

[0066] In this embodiment, the incremental training adopts a flexible parameter update strategy, which divides the network parameters into geometric related parameters θ g Appearance-related parameters θ a The update process prioritizes adjusting appearance parameters to quickly adapt to lighting changes, and then activates geometric parameter updates when structural changes are detected: where Δ g It represents the geometric change amount, which is calculated by the surface normal difference, τ is the geometric change threshold, and η is the learning rate.

[0067] The module includes a historical data management unit and maintains a dynamic memory library through an importance sampling strategy. Define the sample importance weight: in, is the set of historical samples, and β is the decay coefficient. This strategy ensures that the key fields are retained in the memory bank. Diverse samples of the state prevent catastrophic forgetting.

[0068] In this embodiment, the dynamic update process introduces motion consistency constraints and establishes a physical rationality verification mechanism for scene changes through the device motion prior provided by inertial measurement data. Define the residual after motion compensation: Among them, T imu is the pose transformation matrix obtained by IMU integration, P model and P obs The constraint effectively distinguishes the apparent difference caused by the change of the real field and the movement of the device.

[0069] Through the above technical solution, the dynamic update module realizes the continuous optimization of the exhibition hall's three-dimensional model and the precise tracking of scene evolution. Its flexible parameter update strategy and physical constraint mechanism can effectively adapt to dynamic changes such as exhibit replacement and layout adjustment while ensuring model stability.

[0070] The semantic fusion module injects semantic information in stages during the geometric modeling process to generate a semantically queryable 3D model; In this embodiment, the semantic fusion module deeply embeds semantic information into the 3D modeling process through a three-level progressive injection strategy, achieving multi-level collaborative optimization of geometric, texture, and semantic features. This module receives semantic segmentation results and knowledge graph data from the preprocessing module and generates a 3D model with queryable semantic attributes through feature space mapping, attention-guided sampling, and rendering supervision.

[0071] The geometric feature embedding stage uses a graph convolutional network (GCN) to encode semantic labels into 3D mesh vertex features. Define the vertex feature update formula: in: is the feature vector representing the vertex v at the first layer, is the set of adjacent nodes of vertex v, W (l) and b (l) is a learnable parameter and σ is the ReLU activation function.

[0072] This process converts semantic information such as exhibit category and material type into a continuous representation of geometric feature space and establishes a semantic-geometric association mapping.

[0073] In this embodiment, the attention-guided sampling mechanism adjusts the ray sampling distribution by semantic importance weight. The sampling probability function is defined as: Where: q is the semantic query vector, which comes from the embedded representation of the knowledge graph, k(r) is the geometric feature projection of ray r, and d is the feature dimension.

[0074] This mechanism enables the sampling process to prioritize areas with high semantic value (such as exhibit signboards, artwork details, etc.), thereby improving the modeling accuracy of key areas.

[0075] The pixel-level semantic consistency loss is introduced in the rendering supervision stage to force the rendering result to be aligned with the input semantic segmentation map. The loss function is defined as: Where: p is the pixel index, c is the semantic category, y p,c is the one-hot encoding of the true semantic label, To render the semantic probability distribution of the output, the loss is back-propagated through the differentiable rendering pipeline to jointly optimize the geometric reconstruction and semantic segmentation accuracy.

[0076] In this embodiment, the knowledge graph integration module constructs a semantic relationship network for the exhibition hall scene, where nodes represent exhibit entities and edges represent spatiotemporal relationships (such as "adjacent to", "same era", etc.). The semantic weights are dynamically adjusted through the graph attention mechanism: Where: h i is the feature vector of exhibit i, W is the projection matrix, and a is the attention mechanism parameter.

[0077] The semantic association weights output by this module are used to guide the semantic query and interaction function implementation of the scene model.

[0078] Through the above technical solutions, the semantic fusion module realizes the deep coupling of semantic information and geometric modeling process, giving the three-dimensional model interpretable semantic attributes and providing a rich semantic interaction foundation for the digital application of exhibition halls.

[0079] A lightweight output module that converts the optimized radiation field model into a compact representation that supports real-time rendering and outputs it to a visualization terminal through a standardized interface; In this embodiment, the lightweight output module converts the high-precision neural radiation field model into an efficient 3D representation suitable for a variety of devices through layered compression and adaptive rendering techniques. This module receives the optimized scene model from the dynamic update module and performs feature extraction, data compression, and cross-platform adaptation operations, ultimately generating lightweight scene data that supports real-time interaction.

[0080] The edge-cloud collaborative architecture adopts a feature decoupling strategy and deploys a lightweight feature extraction network on edge computing nodes: Where: I is the input image data, H′×W′ is the size of the feature map after downsampling, and C is the number of feature channels. The cloud server receives the edge features and performs neural radiance field inference: (σ,c)=f cloud (F edge ,d,x) Where d is the viewing direction vector and x is the spatial coordinate. This division of labor strategy effectively reduces the computing load on the client side while maintaining the integrity of the cloud model.

[0081] In this embodiment, the vector quantization compression algorithm realizes efficient encoding of feature data by training the codebook. Contains K prototype vectors, and the feature compression process is: Where: f i is the original eigenvector, q(·) is the nearest neighbor quantization function, and δ(·) is the indicator function.

[0082] The codebook is optimized through end-to-end training: Where sg[·] indicates the stop gradient operation, and β is a weight coefficient that balances the reconstruction error and the codebook update strength. The module includes an adaptive level of detail (LOD) generation mechanism that dynamically adjusts the model complexity based on the terminal device performance. The LOD level generation function is defined as: in: is the density gradient amplitude of block i, ∈ is the detail threshold parameter. High gradient areas retain more detail levels, while flat areas use simplified representation to achieve a balance between rendering quality and computing resources.

[0083] In this embodiment, the WebGL rendering engine uses a progressive streaming loading strategy to divide the scene into spatial blocks and multi-resolution texture sets. Define the block loading priority: Among them: B i is the block space range, v is the current viewpoint position, S i is the semantic importance score, and α is the weight coefficient. This strategy prioritizes loading the center area of ​​the field of view and blocks with high semantic value to ensure a smooth experience during user interaction.

[0084] Through the above technical solution, the lightweight output module realizes the complete path of large-scale exhibition hall models from high-precision representation in the cloud to efficient terminal rendering. Its layered compression strategy and adaptive rendering mechanism meet the real-time interaction needs of multi-platform terminals while maintaining visual fidelity.

[0085] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. The exhibition hall 3D modeling intelligent optimization system based on multimodal data fusion is characterized by: include: a data acquisition module configured to simultaneously acquire lidar point cloud data, multispectral image sequences, and inertial measurement unit data; A preprocessing module receives the output of the data acquisition module, aligns the multi-source sensor coordinate system through a spatiotemporal calibration algorithm, and uses a dynamic segmentation network to separate static scenes from dynamic interference elements; The multimodal fusion module receives preprocessed data and adaptively weights the lidar geometric features and visual texture features through a cross-modal attention mechanism; The neural radiation field modeling module constructs a block-based neural radiation field based on the fused features and maintains the geometric continuity between blocks through interface consistency constraints; The reflection-radiation coupling module injects material reflection property parameters into the radiation field rendering pipeline and establishes bidirectional differential constraints to optimize surface material performance; Dynamic update module, triggers incremental training based on scene change detection results and updates neural radiation field parameters; The semantic fusion module injects semantic information in stages during the geometric modeling process to generate a semantically queryable 3D model; A lightweight output module converts the optimized radiation field model into a compact representation that supports real-time rendering and outputs it to the visualization terminal through a standardized interface.

2. The exhibition hall 3D modeling intelligent optimization system based on multimodal data fusion according to claim 1 is characterized in that: The data acquisition module includes: The LiDAR collects millimeter-level precision point cloud data at a frequency of 10 Hz; The multispectral camera synchronously acquires RGB-D image streams with a resolution of no less than 1920×1080; The inertial measurement unit records six-degree-of-freedom posture data at a frequency of 90Hz-120Hz.

3. The exhibition hall 3D modeling intelligent optimization system based on multimodal data fusion according to claim 1 is characterized in that: The spatiotemporal calibration algorithm of the preprocessing module solves the nine-degree-of-freedom transformation matrix by minimizing the reprojection error: T=argmin∑||π(Rp i +t)-u i || 2 ; Where R is the rotation matrix, t is the translation vector, π(·) is the camera projection model, and p is the ray sampling probability distribution function.

4. The exhibition hall 3D modeling intelligent optimization system based on multimodal data fusion according to claim 3 is characterized in that: The dynamic segmentation network adopts a two-stream architecture: The static branch uses 3D sparse convolution to process point cloud data, and the dynamic branch extracts motion features through the optical flow network, and finally generates a dynamic mask through a gating mechanism.

5. The exhibition hall 3D modeling intelligent optimization system based on multimodal data fusion according to claim 1 is characterized in that: The cross-modal attention mechanism of the multimodal fusion module satisfies: Among them, the query vector Q comes from the lidar feature, and the key-value pairs K and V are generated by splicing visual features.

6. The exhibition hall 3D modeling intelligent optimization system based on multimodal data fusion according to claim 1 is characterized in that: The block strategy of the neural radiation field modeling module divides the scene into N×N independent modeling blocks and imposes interface constraints: in, is a set of adjacent block pairs, x ij The interface sampling point.

7. The exhibition hall 3D modeling intelligent optimization system based on multimodal data fusion according to claim 1 is characterized in that: The differential constraints established by the reflection-radiation coupling module include: Among them, f r is the bidirectional reflectance distribution function based on the microsurface model.

8. The exhibition hall 3D modeling intelligent optimization system based on multimodal data fusion according to claim 1 is characterized in that: The incremental training of the dynamic update module adopts an adaptive learning rate strategy: or t =η0·exp(-α·t); Among them, α is the decay coefficient and t is the number of update iterations.

9. The exhibition hall 3D modeling intelligent optimization system based on multimodal data fusion according to claim 1 is characterized in that: The phased injection of the semantic fusion module includes: The geometric embedding stage encodes semantic labels into mesh vertex features; The sampling stage adjusts the ray distribution through attention weights; A pixel-level semantic consistency loss is imposed during the rendering phase.

10. The exhibition hall 3D modeling intelligent optimization system based on multimodal data fusion according to claim 1 is characterized in that: The lightweight output module is: Edge computing nodes perform feature extraction and data compression; The cloud server performs radiation field calculations and parameter updates; The WebGL engine enables real-time rendering on the browser side.

Images