Dynamic modeling method of scene space 3D model based on multimodal data

Abstract

An embodiment of the present application provides a method for dynamic modeling of a scene space three-dimensional model based on multimodal data, which establishes a spatiotemporal correspondence by fusing the feature information of video sequences, point cloud data, and IMU data using a particle filter algorithm. A Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates scene scores based on spatial density, depth occlusion, and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a method for dynamic modeling of a scene space three-dimensional model based on multimodal data. Background Art

[0002] Traditional 3D scene reconstruction methods face numerous challenges when handling large-scale, complex scenes. Existing single-source data modeling solutions struggle to cope with complex situations such as occlusion, changing lighting, and dynamic objects. Simple multi-source data fusion methods also fail to fully leverage the complementary strengths of various sensor data types. While the introduction of deep learning technology has improved modeling performance, existing systems still suffer from poor model adaptability and low reconstruction efficiency.

[0003] Especially when dealing with large-scale, complex indoor and outdoor scenes, traditional methods struggle to accurately identify and process dynamic areas, and fail to effectively balance reconstruction accuracy and computational efficiency. Furthermore, fixed modeling strategies cannot dynamically adjust to scene characteristics, resulting in significant differences in reconstruction quality across different scene types. Furthermore, existing systems lack a deep understanding and utilization of scene semantics, making it difficult to implement intelligent modeling parameter configuration.

[0004] Key challenges currently in need of resolution include achieving deep fusion of multimodal data, establishing accurate representations of scene features, building intelligent modeling strategy selection mechanisms, and improving the system's adaptability to dynamic scenes. These challenges not only impact the quality and efficiency of 3D reconstruction but also form the foundation for intelligent scene understanding and modeling. Summary of the Invention

[0005] In response to the problems in the existing technology, the present application provides a dynamic modeling method of scene space three-dimensional model based on multimodal data, which can effectively improve the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0006] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a method for dynamic modeling of a scene space three-dimensional model based on multimodal data, comprising:

[0008] Collect video sequences, point cloud data, and IMU data of large-scale indoor and outdoor scenes, perform deep learning noise reduction and distortion correction on the video sequences, perform spatial downsampling and denoising on the point cloud data, perform zero-bias calibration and calibration on the IMU data, extract ORB image features and depth information from the video sequences, extract normal vector features and edge features from the point cloud data, extract acceleration sequences and angular velocity sequences from the IMU data, establish spatiotemporal correspondences among the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence, and the angular velocity sequence, and use a particle filter algorithm to fuse these correspondences to obtain multi-source scene information;

[0009] Constructing a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain scene semantic types, performing octree partitioning on the point cloud data to calculate spatial density coefficients, using a frustum projection method to statistically calculate a depth occlusion map of the multi-source scene information to obtain a scene complexity coefficient, constructing a geometric descriptor based on the normal vector features and the edge features to calculate a shape variation coefficient, nonlinearly weighting the density coefficient, the complexity coefficient, and the shape variation coefficient to obtain a scene score, constructing a multimodal feature representation based on the scene semantic type and the scene score, and selecting a modeling algorithm and parameter combination in a policy network trained by deep reinforcement learning;

[0010] A deep neural network is used to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, the topological structure change of the point cloud data is calculated to generate a deformation vector field, the camera motion trajectory is estimated based on the fusion of the acceleration sequence and the angular velocity sequence, the three-dimensional motion field, the deformation vector field and the motion trajectory are input into a graph neural network to predict the dynamic area of ​​the scene, the modeling algorithm and the parameter combination are applied to the dynamic area to perform adaptive mesh subdivision and geometric constraint reconstruction, a dynamic threshold is calculated according to the semantic type of the scene, and when the reconstruction error of the dynamic area exceeds the dynamic threshold, the modeling algorithm is reselected in the strategy network to reconstruct the dynamic area.

[0011] Furthermore, the performing deep learning noise reduction and distortion correction on the video sequence, performing spatial downsampling and denoising on the point cloud data, performing zero bias calibration and calibration on the IMU data, extracting ORB image features and depth information from the video sequence, extracting normal vector features and edge features from the point cloud data, and extracting acceleration sequences and angular velocity sequences from the IMU data includes:

[0012] Inputting the video sequence into a U-shaped convolutional neural network for multi-scale feature extraction, performing skip connection fusion on the feature map to obtain a noise distribution map, reconstructing a denoised video sequence based on the noise distribution map, calculating the positions of checkerboard corner points in the video sequence, constructing a nonlinear equation system based on the Zhang camera model, solving the equation system using the Levenberg-Marquardt algorithm to obtain lens distortion parameters, and correcting the denoised video sequence according to the distortion parameters;

[0013] FAST corner detection is performed on the video sequence to obtain a set of feature points, the BRIEF descriptor of the feature point set is calculated to construct ORB features, a binocular disparity algorithm is used to calculate a depth map, a K-nearest neighbor graph is constructed for the point cloud data, the covariance matrix of each point is calculated based on the K-nearest neighbor graph to obtain a normal vector, the edge detection operator is used to extract point cloud edge features, the IMU data is bandpass filtered to obtain an acceleration sequence, and the IMU data is subjected to gyroscope integration to obtain an angular velocity sequence.

[0014] Furthermore, establishing a spatiotemporal correspondence between the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence, and the angular velocity sequence, and fusing the correspondence using a particle filter algorithm to obtain multi-source scene information includes:

[0015] A KD tree index structure is constructed for the ORB image features, feature matching pairs are calculated based on nearest neighbor search, abnormal matching points are eliminated using the RANSAC algorithm, the depth information is projected into three-dimensional space to establish a point cloud mapping relationship, the normal vector features and the edge features are spatially indexed, a feature association table with timestamp alignment is established, and a feature spatiotemporal consistency verification model is constructed using camera pose and IMU motion constraints;

[0016] A state space equation based on the feature spatiotemporal consistency verification model is constructed, the particle swarm state vector and weight are initialized, the likelihood function between the observation data in the feature association table and the particle state is calculated, the particle weight is updated according to the likelihood function, a resampling process is performed to eliminate weight degradation, and the fused multi-source scene information is obtained based on the weighted average of the particle swarm state vector.

[0017] Furthermore, the method of constructing a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain scene semantic types, performing octree partitioning on the point cloud data to calculate spatial density coefficients, using a frustum projection method to calculate a depth occlusion map of the multi-source scene information to obtain a scene complexity coefficient, and constructing a geometric descriptor based on the normal vector features and the edge features to calculate a shape variation coefficient includes:

[0018] Mapping the multi-source scene information into a feature sequence, position-encoding the feature sequence, calculating the association weights between features through a multi-head self-attention mechanism, transforming the weighted features using a feedforward neural network, constructing a semantic segmentation head network to classify scene elements, and generating a scene semantic type label map;

[0019] The point cloud space is recursively divided into octrees, the number of point clouds in each leaf node is counted to calculate the density coefficient, a camera frustum model is constructed to project the scene, the projection overlapping area is calculated to obtain the complexity coefficient, the normal vector feature and the edge feature are combined to construct a local geometric descriptor, and the variance of the descriptor is calculated to obtain the shape change coefficient.

[0020] Furthermore, the density coefficient, the complexity coefficient, and the shape change coefficient are nonlinearly weighted to obtain a scene score, a multimodal feature representation is constructed according to the scene semantic type and the scene score, and a modeling algorithm and parameter combination are selected in a policy network trained by deep reinforcement learning, including:

[0021] Performing feature mapping on the density coefficient, the complexity coefficient, and the shape change coefficient using a multi-layer perceptron network, inputting the mapping result into a Softmax layer to calculate a weight coefficient, performing a weighted sum of the three coefficients based on the weight coefficient to obtain a scene score, performing one-hot encoding on the scene semantic type, and concatenating the scene score and the encoding result to construct a multimodal feature vector;

[0022] The multimodal feature vector is used as the state input of the policy network, an action space including modeling algorithm selection and parameter adjustment is constructed, the policy gradient algorithm is used to optimize the network parameters, a reward function is designed based on the reconstruction accuracy, the modeling algorithm is selected by exploring and utilizing the balance strategy, and the corresponding parameter combination is determined according to the reward function.

[0023] Furthermore, the method of using a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculating the topological structure change of the point cloud data to generate a deformation vector field, and estimating the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence includes:

[0024] Input the video sequence and the depth information into a three-dimensional convolutional neural network to extract spatiotemporal features, restore the scene spatial resolution through a deconvolution layer, generate a dense optical flow field, construct a topological adjacency graph for the point cloud data, calculate the displacement difference of point cloud nodes at adjacent time points, and generate a vector field describing the local deformation;

[0025] The acceleration sequence is gravity compensated and integrated to calculate the displacement, the angular velocity sequence is integrated to obtain the rotation quaternion, an extended Kalman filter state equation is constructed, the displacement and the rotation quaternion are used as observation quantities, and the six-degree-of-freedom pose parameters of the camera are recursively estimated based on the kinematic model to generate the camera motion trajectory.

[0026] Furthermore, the three-dimensional motion field, the deformation vector field, and the motion trajectory are input into a graph neural network to predict a dynamic region of a scene, the modeling algorithm and the parameter combination are applied to the dynamic region to perform adaptive mesh subdivision and geometric constraint reconstruction, a dynamic threshold is calculated according to the scene semantic type, and when the reconstruction error of the dynamic region exceeds the dynamic threshold, a modeling algorithm is reselected in the strategy network to reconstruct the dynamic region, including:

[0027] Constructing a graph convolutional network to aggregate features of the three input fields, capturing temporal variation features through a gated recurrent unit, and using a graph pooling layer to output a probability map of the dynamic region. Based on the probability map, the boundaries of the dynamic region are determined, quadtree meshing is performed within the dynamic region, and the mesh nodes are geometrically reconstructed using the modeling algorithm.

[0028] A baseline threshold is set according to the scene semantic type, the baseline threshold is dynamically adjusted based on the scene complexity, the geometric error between the reconstructed point cloud and the original point cloud of the dynamic area is calculated, the geometric error is compared with the dynamic threshold, and the policy network is triggered to re-execute the modeling algorithm selection.

[0029] In a second aspect, the present application provides a device for dynamically modeling a scene space three-dimensional model based on multimodal data, comprising:

[0030] A multi-source data processing module is used to collect video sequences, point cloud data, and IMU data of large-scale indoor and outdoor scenes, perform deep learning noise reduction and distortion correction on the video sequences, perform spatial downsampling and denoising on the point cloud data, perform zero-bias calibration and calibration on the IMU data, extract ORB image features and depth information from the video sequences, extract normal vector features and edge features from the point cloud data, extract acceleration sequences and angular velocity sequences from the IMU data, establish spatiotemporal correspondences among the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence, and the angular velocity sequence, and fuse the correspondences using a particle filter algorithm to obtain multi-source scene information;

[0031] A scene space processing module is used to construct a Transformer-based multimodal scene analysis network, perform instance segmentation on the multi-source scene information to obtain scene semantic types, perform octree partitioning on the point cloud data to calculate spatial density coefficients, use a frustum projection method to statistically calculate a depth occlusion map of the multi-source scene information to obtain a scene complexity coefficient, construct a geometric descriptor based on the normal vector features and the edge features to calculate a shape variation coefficient, nonlinearly weight the density coefficient, the complexity coefficient, and the shape variation coefficient to obtain a scene score, construct a multimodal feature representation based on the scene semantic type and the scene score, and select a modeling algorithm and parameter combination in a policy network trained by deep reinforcement learning;

[0032] A dynamic modeling module is used to use a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculate the topological structure changes of the point cloud data to generate a deformation vector field, estimate the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence, input the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, apply the modeling algorithm and the parameter combination to the dynamic area for adaptive mesh subdivision and geometric constraint reconstruction, calculate a dynamic threshold according to the semantic type of the scene, and reselect the modeling algorithm in the strategy network to reconstruct the dynamic area when the reconstruction error of the dynamic area exceeds the dynamic threshold.

[0033] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data are implemented.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data.

[0035] In a fifth aspect, the present application provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data.

[0036] It can be seen from the above technical solution that the present application provides a method for dynamic modeling of a scene space three-dimensional model based on multimodal data, which establishes a spatiotemporal correspondence by fusing the feature information of video sequences, point cloud data and IMU data, and adopts a particle filter algorithm. The Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates scene scores by combining spatial density, depth occlusion and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 This is a flow chart of a method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application;

[0039] Figure 2 This is a second flow chart of a method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application;

[0040] Figure 3 This is a third flow chart of a method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application;

[0041] Figure 4 This is a fourth flow chart of a method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application;

[0042] Figure 5 This is a fifth flow chart of a method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application;

[0043] Figure 6 This is a sixth flow chart of a method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application;

[0044] Figure 7 This is a seventh flow chart of a method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application;

[0045] Figure 8 4 is a structural diagram of a dynamic modeling device for a scene space three-dimensional model based on multimodal data in an embodiment of the present application;

[0046] Figure 9 Schematic diagram of the structure of the electronic device in the embodiment of the present application.

[0047] Reference numerals:

[0048] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0049] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] The acquisition, storage, use, and processing of data in this application's technical solution comply with relevant national laws and regulations.

[0051] Taking into account the problems existing in the prior art, the present application provides a method for dynamic modeling of scene space three-dimensional models based on multimodal data, which establishes spatiotemporal correspondences by fusing feature information of video sequences, point cloud data and IMU data, and adopts a particle filter algorithm. A Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates scene scores by combining spatial density, depth occlusion and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0052] In order to effectively improve the quality and efficiency of 3D reconstruction of large-scale complex scenes, this application provides an embodiment of a method for dynamic modeling of scene space 3D models based on multimodal data, see Figure 1 The method for dynamic modeling of a scene space three-dimensional model based on multimodal data specifically includes the following contents:

[0053] Step S101: collecting video sequences, point cloud data, and IMU data of large-scale indoor and outdoor scenes, performing deep learning noise reduction and distortion correction on the video sequences, performing spatial downsampling and denoising on the point cloud data, performing zero-bias calibration and calibration on the IMU data, extracting ORB image features and depth information from the video sequences, extracting normal vector features and edge features from the point cloud data, extracting acceleration sequences and angular velocity sequences from the IMU data, establishing spatiotemporal correspondences among the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequences, and the angular velocity sequences, and fusing the correspondences using a particle filter algorithm to obtain multi-source scene information;

[0054] Optionally, this embodiment proposes a multi-source data acquisition and preprocessing method, which achieves high-quality acquisition and feature extraction of scene information by fusing video, point cloud and IMU data.

[0055] This example first uses professional-grade cameras to capture high-resolution video sequences, uses lidar to acquire high-precision point cloud data, and uses an IMU sensor to record motion parameters. During the data acquisition process, the three devices are synchronized and spatially aligned to establish a unified data acquisition reference system.

[0056] This embodiment uses an improved U-Net network structure for denoising video sequences. This network accurately identifies image noise distribution through multi-level feature extraction and fusion, achieving adaptive noise reduction. For distortion correction, an improved Zhang calibration method is employed. Checkerboard corner detection is used to establish a nonlinear system of equations, and the LM algorithm is used to solve for camera intrinsic parameters and distortion coefficients.

[0057] Point cloud data preprocessing utilizes an adaptive voxel filtering method. This embodiment dynamically adjusts voxel size based on the point cloud density distribution, achieving data downsampling while preserving geometric detail. Statistical outlier analysis and local surface fitting effectively remove noise points and outliers.

[0058] IMU data processing focuses on resolving the problem of bias accumulation. This embodiment designs a bias estimation algorithm based on static detection, calculating the bias value by identifying static segments. The calibration process uses a multi-pose optimization method to establish a mapping relationship between the IMU output and the reference motion.

[0059] In the feature extraction phase, this embodiment extracts ORB features from video sequences, including FAST corner detection and BRIEF descriptor calculation. Depth information is obtained through an improved binocular matching algorithm, improving the accuracy of depth estimation. Point cloud feature extraction includes normal vector calculation based on principal curvature analysis and edge feature extraction based on curvature changes.

[0060] A multi-level matching strategy is used to establish spatiotemporal correspondence. This embodiment first establishes spatial correspondence between features, accelerating nearest neighbor search using a KD tree. Temporal correspondence is achieved using a sliding window approach, accounting for sensor latency and sampling frequency variations. Feature association employs the RANSAC algorithm to eliminate false matches and improve the reliability of correspondences.

[0061] Data fusion utilizes an improved particle filtering algorithm. This embodiment designs a state transition model and observation model that adapts to the characteristics of multi-source data, mitigating particle degradation through an adaptive resampling strategy. The filtering process considers the uncertainty of various features and performs a weighted fusion of information from different sources.

[0062] In practical applications, this embodiment achieves accurate geometric reconstruction and texture mapping for indoor scene reconstruction. In outdoor environments, it can adapt to lighting changes and occlusions while maintaining stable feature extraction. For dynamic scenes, multi-source data fusion improves the tracking accuracy of moving targets.

[0063] This embodiment addresses the incomplete information problem of traditional single-source data collection methods. Multi-source data fusion improves the comprehensiveness of scene perception, and efficient preprocessing and feature extraction ensure data quality for subsequent modeling. This solution is particularly suitable for applications requiring precise 3D reconstruction, such as indoor navigation and augmented reality.

[0064] Step S102: constructing a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain a scene semantic type, performing octree partitioning on the point cloud data to calculate a spatial density coefficient, using a frustum projection method to statistically calculate a depth occlusion map of the multi-source scene information to obtain a scene complexity coefficient, constructing a geometric descriptor based on the normal vector features and the edge features to calculate a shape variation coefficient, performing nonlinear weighting on the density coefficient, the complexity coefficient, and the shape variation coefficient to obtain a scene score, constructing a multimodal feature representation based on the scene semantic type and the scene score, and selecting a modeling algorithm and parameter combination in a policy network trained by deep reinforcement learning;

[0065] Optionally, this embodiment proposes a scene evaluation method based on multimodal analysis, which realizes adaptive evaluation of scene complexity and intelligent selection of modeling strategies by fusing semantic information and geometric features.

[0066] In constructing a multimodal scene analysis network, this embodiment uses an improved Transformer architecture and a multi-head attention mechanism to process feature information from different modalities. The network inputs include image features, point cloud features, and motion feature sequences, with position encoding preserving spatial position information. The attention layer calculates the correlation weights between different features, enabling effective cross-modal feature fusion.

[0067] Scene semantic analysis uses instance segmentation. This embodiment integrates a segmentation head network after the Transformer decoder, improving segmentation accuracy through a multi-scale feature pyramid structure. The network output includes category labels and instance masks for scene elements, enabling differentiation between different semantic types such as walls, floors, and furniture.

[0068] Spatial density assessment uses an adaptive octree partitioning strategy. This embodiment dynamically adjusts the partitioning granularity based on the local distribution characteristics of the point cloud, accurately representing scene details while maintaining computational efficiency. The density coefficient is normalized to reflect the geometric complexity of the scene.

[0069] Scene complexity assessment uses an improved frustum projection method. This embodiment constructs a multi-view projection model to calculate the overlap and occlusion relationships of scene elements. Depth map analysis identifies visually occluded areas and assesses the scene's spatial layout complexity.

[0070] Shape change analysis is based on local geometric features. This embodiment combines normal vectors and edge features to construct a multi-scale geometric descriptor, describing local shape changes through principal direction analysis and curvature statistics. The variance of the descriptor reflects the degree of geometric irregularity in the scene.

[0071] Scenario scoring utilizes a nonlinear weighted fusion approach. This embodiment designs a multilayer perceptron network to learn the nonlinear relationship between the three evaluation coefficients. A softmax layer calculates adaptive weights, enabling dynamic fusion of evaluation metrics. The scoring results comprehensively reflect the modeling difficulty of the scenario.

[0072] Multimodal feature representation uses a hybrid encoding approach. This embodiment uses one-hot encoding of scene semantic types and concatenates them with scene score features to construct a joint representation. This representation preserves both the semantic and complexity information of the scene, providing a basis for subsequent modeling strategy selection.

[0073] The modeling strategy selection uses deep reinforcement learning. This example constructs an action space that includes algorithm selection and parameter adjustment, and designs a reward function based on reconstruction accuracy. The network parameters are optimized using a policy gradient algorithm, enabling online learning and optimization of the modeling strategy.

[0074] In practical applications, this embodiment can adaptively select the appropriate modeling strategy based on the characteristics of the scene. For regular indoor scenes, a fast reconstruction algorithm based on planar features is preferred. For complex outdoor environments, a fine reconstruction method with stronger geometric constraints is selected.

[0075] This embodiment addresses the difficulty of traditional fixed modeling strategies in adapting to complex scenarios. Through multi-dimensional scene evaluation and intelligent strategy selection, it improves the adaptability and robustness of 3D reconstruction. This solution demonstrates excellent modeling results in applications such as large-scale scene reconstruction and augmented reality.

[0076] Step S103: Using a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculating the topological structure change of the point cloud data to generate a deformation vector field, estimating the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence, inputting the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, applying the modeling algorithm and the parameter combination to the dynamic area to perform adaptive mesh subdivision and geometric constraint reconstruction, calculating a dynamic threshold according to the semantic type of the scene, and reselecting the modeling algorithm in the strategy network to reconstruct the dynamic area when the reconstruction error of the dynamic area exceeds the dynamic threshold.

[0077] Optionally, this embodiment proposes a dynamic scene adaptive reconstruction method, which achieves accurate three-dimensional reconstruction of complex dynamic scenes through multi-source motion information analysis and intelligent algorithm selection.

[0078] For scene motion analysis, this embodiment employs a two-stream neural network architecture to process RGB image sequences and depth map sequences separately. The spatial stream network extracts spatial features of the scene, while the temporal stream network captures temporal variations. A feature fusion layer combines the information from these two branches to generate a three-dimensional vector field describing the scene motion.

[0079] Point cloud deformation analysis uses a local structure tracking method. This embodiment constructs a local connectivity graph for the point cloud and calculates the neighborhood topology of each point. Based on the structural changes between consecutive time frames, the local deformation of the point cloud is estimated. The deformation vector field reflects the non-rigid deformation characteristics of objects in the scene.

[0080] Camera motion estimation uses a tightly coupled inertial-visual fusion approach. This example combines the acceleration and angular velocity information from IMU data with visual feature matching results to construct a nonlinear optimization problem. Using a sliding window optimization method, accurate real-time camera pose estimation is achieved.

[0081] Dynamic region prediction uses a graph neural network model. This embodiment encodes the 3D motion field, deformation vector field, and camera trajectory into node features of a graph structure. Dynamic features are extracted using graph convolutional layers and an attention mechanism. The network outputs a probability map of the dynamic region, which guides subsequent adaptive reconstruction.

[0082] Adaptive mesh subdivision is based on an octree structure. This embodiment adaptively adjusts the mesh granularity based on the spatial distribution and motion characteristics of dynamic areas. A finer mesh is used in areas with intense motion to improve reconstruction accuracy.

[0083] Geometrically constrained reconstruction uses a multi-view stereo vision approach. This embodiment combines the modeling algorithms and parameters selected in the previous steps to construct geometric constraints that take temporal consistency into account. By minimizing the projection error and regularization term, the geometric accuracy of the reconstruction results is optimized.

[0084] Dynamic threshold calculation takes into account scene semantics. This embodiment designs an adaptive threshold calculation strategy based on the geometric features and motion characteristics of different semantic types. For example, a more relaxed threshold is used for non-rigid objects such as the human body, while a stricter threshold is used for rigid objects such as buildings.

[0085] Reconstruction quality is assessed using multi-scale error analysis. This embodiment evaluates reconstruction accuracy by comparing the geometric deviations between the reconstruction result and the original point cloud. When the error exceeds a dynamic threshold, the policy network is triggered to reselect a more appropriate modeling algorithm.

[0086] In practical applications, this embodiment can effectively handle complex scenes containing multiple moving objects. For human motion in indoor scenes, it can accurately reconstruct motion postures. For vehicle motion in outdoor scenes, it can precisely capture motion trajectories and shape changes.

[0087] This embodiment addresses the problem of traditional static reconstruction methods failing in dynamic scenes. By using multi-source motion analysis and an adaptive reconstruction strategy, the accuracy and robustness of dynamic scene reconstruction are improved. This solution is of great value in applications requiring real-time scene understanding, such as robotic navigation and augmented reality.

[0088] From the above description, it can be seen that the dynamic modeling method of the scene space three-dimensional model based on multimodal data provided in the embodiment of the present application can establish a spatiotemporal correspondence by fusing the feature information of video sequences, point cloud data and IMU data using a particle filter algorithm. The Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates scene scores based on spatial density, depth occlusion and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0089] In one embodiment of the method for dynamic modeling of scene space three-dimensional model based on multimodal data of the present application, see Figure 2 , and can also include the following:

[0090] Step S201: Inputting the video sequence into a U-shaped convolutional neural network for multi-scale feature extraction, performing skip connection fusion on the feature map to obtain a noise distribution map, reconstructing the denoised video sequence based on the noise distribution map, calculating the positions of checkerboard corner points in the video sequence, constructing a nonlinear equation system based on the Zhang camera model, solving the equation system using the Levenberg-Marquardt algorithm to obtain lens distortion parameters, and correcting the denoised video sequence according to the distortion parameters;

[0091] Step S202: Perform FAST corner detection on the video sequence to obtain a feature point set, calculate the BRIEF descriptor of the feature point set to construct ORB features, use the binocular disparity algorithm to calculate the depth map, construct a K-nearest neighbor graph for the point cloud data, calculate the covariance matrix of each point based on the K-nearest neighbor graph to obtain the normal vector, use the edge detection operator to extract the point cloud edge features, perform bandpass filtering on the IMU data to obtain an acceleration sequence, and perform gyroscope integration on the IMU data to obtain an angular velocity sequence.

[0092] Optionally, this embodiment proposes a multi-sensor data preprocessing and feature extraction method, which achieves high-quality visual, point cloud and inertial data processing by combining deep learning and geometric analysis.

[0093] This embodiment uses an improved U-Net network structure for video sequence denoising. This network consists of two main components: an encoder and a decoder. The encoder extracts multi-scale features through multi-layer convolution and pooling operations, while the decoder gradually restores image resolution through deconvolution. During feature fusion, a skip connection structure is used to directly transfer low-level features from the encoder to the corresponding layers of the decoder, preserving image detail. The network outputs a noise distribution map, which is used to guide the image denoising process.

[0094] Camera calibration uses a modified Zhang calibration method. This embodiment first detects checkerboard corners in the video sequence and improves position accuracy using a sub-pixel precision corner detection algorithm. A mapping relationship between the world coordinate system and the image coordinate system is established based on the homography matrix, and a nonlinear equation system containing camera intrinsic parameters and distortion parameters is constructed. The Levenberg-Marquardt optimization algorithm is used to solve the equation system and obtain the radial and tangential distortion parameters of the lens.

[0095] Image feature extraction uses the ORB (Oriented FAST and Rotated BRIEF) algorithm. This example first uses the FAST (Features from Accelerated Segment Test) algorithm to detect corners and selects the feature points with the strongest response values ​​through non-maximum suppression. The principal direction is calculated for each feature point to ensure rotational invariance. The BRIEF descriptor generates a binary string by comparing the grayscale values ​​of pixels in the neighborhood of a feature point, providing a compact and computationally efficient feature representation.

[0096] Depth information is acquired using an improved binocular matching algorithm. This embodiment calculates the disparity map using a local stereo matching method and optimizes the disparity estimation results using dynamic programming. This considers image texture and edge information to improve depth estimation accuracy in weakly textured areas.

[0097] Point cloud feature extraction involves calculating normal vectors and edge features. This example first constructs a local connectivity graph based on K-nearest neighbors and calculates the covariance matrix for each point using principal component analysis. The normal vector is determined by the eigenvector corresponding to the minimum eigenvalue of the covariance matrix. Edge features are extracted by analyzing local curvature changes, effectively identifying geometrically discontinuous areas in the point cloud.

[0098] IMU data processing utilizes a multi-stage filtering strategy. This embodiment applies bandpass filtering to the acceleration data to remove high-frequency noise and low-frequency drift. Angular velocity data is acquired using an improved numerical integration method, employing zero-bias estimation and temperature compensation to improve integration accuracy.

[0099] In indoor navigation scenarios, the data preprocessing method of this embodiment can effectively improve the stability of feature extraction. Through noise reduction and correction, image quality is improved in low-light environments. The point cloud feature extraction method demonstrates good robustness when processing reflective surfaces.

[0100] This embodiment addresses the issue of unstable multi-source data quality. By combining deep learning with geometric analysis, it improves the accuracy and robustness of feature extraction. This solution provides a high-quality data foundation for subsequent scene reconstruction and motion estimation, and has broad application value in fields such as augmented reality and robotic vision.

[0101] In one embodiment of the method for dynamic modeling of scene space three-dimensional model based on multimodal data of the present application, see Figure 3 , and can also include the following:

[0102] Step S301: constructing a KD tree index structure for the ORB image features, calculating feature matching pairs based on nearest neighbor search, removing abnormal matching points using the RANSAC algorithm, projecting the depth information into three-dimensional space to establish a point cloud mapping relationship, performing spatial position indexing on the normal vector features and the edge features, establishing a feature association table aligned with timestamps, and constructing a feature spatiotemporal consistency verification model using camera pose and IMU motion constraints;

[0103] Step S302: Construct a state space equation based on the feature spatiotemporal consistency verification model, initialize the particle swarm state vector and weight, calculate the likelihood function between the observation data in the feature association table and the particle state, update the particle weight according to the likelihood function, perform a resampling process to eliminate weight degradation, and obtain the fused multi-source scene information based on the weighted average of the particle swarm state vector.

[0104] Optionally, this embodiment proposes a spatiotemporal correlation and fusion method for multi-source scene information, which achieves efficient fusion of visual, point cloud and inertial data through feature index construction and particle filter optimization.

[0105] During feature matching, this embodiment first constructs a KD tree index structure for ORB features. By establishing a binary tree partition in the high-dimensional feature space, nearest neighbor searches for feature points are accelerated. Feature matching employs a ratio test strategy, selecting reliable matching pairs by setting a distance threshold. The RANSAC algorithm uses random sampling and consistency checks to eliminate anomalous matching points that do not conform to spatial transformation constraints.

[0106] The point cloud mapping relationship is established using a depth projection method. This embodiment projects the depth information of image feature points into three-dimensional space using the camera intrinsic parameter matrix, establishing a correspondence between two-dimensional image features and three-dimensional point clouds. This mapping relationship provides spatial constraints for subsequent feature fusion.

[0107] The geometric feature index uses a multi-level spatial partitioning strategy. This embodiment constructs an octree index structure for the normal vector features and edge features of the point cloud, supporting fast spatial neighborhood queries. By analyzing the spatial distribution characteristics of the features, local connectivity relationships between feature points are established.

[0108] The construction of the feature association table takes time synchronization into account. This embodiment uses hardware timestamps to align data from different sensors and establish a unified time reference. The association table records information such as the timestamp, spatial location, and descriptor of feature points, providing a spatiotemporal index for feature fusion.

[0109] The spatiotemporal consistency verification model incorporates multiple constraints. This embodiment utilizes the camera's motion model and the IMU's inertial constraints to construct a physical model of feature motion. The rationality of feature associations is verified by analyzing the temporal changes and spatial distribution of feature points.

[0110] State-space equations describe the feature fusion process. This embodiment combines state variables such as the position and velocity of feature points into a state vector, constructing a nonlinear state equation that takes into account multi-sensor observations. The observation equation includes multiple constraints, such as visual reprojection error and IMU pre-integration error.

[0111] Particle filter optimization employs an adaptive resampling strategy. This embodiment first initializes a particle swarm, with each particle representing a possible state hypothesis. The weight of each particle is evaluated by calculating the likelihood function between the observed data and the particle state. When the weight distribution degrades, a resampling process is performed to maintain particle diversity.

[0112] In indoor scene reconstruction applications, this embodiment effectively addresses sensor data asynchrony and measurement noise. For fast-moving scenes, IMU constraints improve feature tracking stability. In environments with varying lighting, geometric features provide additional matching constraints, enhancing the robustness of feature association.

[0113] This embodiment solves the problems of inaccurate feature association and unstable state estimation in multi-source data fusion. Through an efficient feature indexing structure and particle filter optimization, the accuracy and real-time performance of feature fusion are improved. This solution demonstrates significant advantages in applications requiring precise scene understanding, such as visual SLAM and augmented reality.

[0114] In one embodiment of the method for dynamic modeling of scene space three-dimensional model based on multimodal data of the present application, see Figure 4 , and can also include the following:

[0115] Step S401: Mapping the multi-source scene information into a feature sequence, position encoding the feature sequence, calculating the correlation weights between features through a multi-head self-attention mechanism, transforming the weighted features using a feedforward neural network, constructing a semantic segmentation head network to classify scene elements, and generating a scene semantic type label map;

[0116] Step S402: recursively divide the point cloud space into octrees, count the number of point clouds in each leaf node to calculate the density coefficient, construct a camera frustum model to project the scene, calculate the projection overlap area to obtain the complexity coefficient, combine the normal vector feature and the edge feature to construct a local geometric descriptor, and calculate the variance of the descriptor to obtain the shape change coefficient.

[0117] Optionally, this embodiment proposes a scene semantic understanding and complexity assessment method based on deep learning, which achieves accurate semantic segmentation and feature description of the scene through feature sequence analysis and multi-dimensional feature fusion.

[0118] During feature sequence processing, this embodiment first converts multi-source scene information into a unified feature representation. Using sinusoidal position encoding, the spatial position information of the features is encoded into a feature vector. The position encoding involves sinusoidal functions of varying frequencies, enabling the network to perceive the relative positional relationships of the features.

[0119] The multi-head self-attention mechanism achieves global feature association. This embodiment feeds the feature sequence into multiple parallel attention heads. Each attention head independently computes the query and key-value matrices, and calculates the association weights between features using scaled dot-product attention. This multi-head mechanism allows the network to learn feature relationships from different representation subspaces, enhancing the expressive power of feature extraction.

[0120] A feedforward neural network performs feature transformation. This embodiment uses a two-layer fully connected network structure, enhancing the network's expressive power through nonlinear activation functions. Residual connections and layer normalization are used to stabilize the training process and improve network convergence performance.

[0121] The semantic segmentation head network adopts a multi-scale feature fusion strategy. This embodiment designs a feature pyramid structure that fuses feature information at different levels through upsampling and skip connections. The decoder gradually restores the feature resolution and ultimately outputs a semantic label map of the same size as the input image.

[0122] Scene complexity assessment uses multidimensional feature analysis. This embodiment first adaptively partitions the point cloud space using a recursive octree. The number of point clouds within each leaf node is counted, and the local point cloud density coefficient is calculated to reflect the scene's geometric complexity.

[0123] View cone projection analysis considers scene visibility. This embodiment constructs a camera view cone model and calculates the overlapping area of ​​the scene's projections from different perspectives. The area and depth changes of the overlapping area are used to assess the scene's observation complexity.

[0124] Local geometry description uses a multi-feature fusion approach. This embodiment combines normal and edge features to construct a descriptor that includes geometric information such as the local curvature and principal curvature direction of the point cloud. By calculating the variance of the descriptor within a local neighborhood, the degree of shape change in the scene is quantified.

[0125] In indoor scene understanding applications, this embodiment can accurately identify different semantic elements such as furniture and walls. For areas with complex geometric structures, such as stairwells and multi-layered bookshelves, it can accurately assess the scene complexity and provide a reference for subsequent reconstruction.

[0126] This embodiment addresses the poor adaptability of traditional scene understanding methods to complex scenarios. By combining deep learning with geometric analysis, the accuracy and robustness of scene semantic understanding are improved. This solution is valuable in applications such as robotic environmental perception and intelligent scene reconstruction, providing reliable semantic information and complexity assessment for subsequent scene modeling.

[0127] In one embodiment of the method for dynamic modeling of scene space three-dimensional model based on multimodal data of the present application, see Figure 5 , and can also include the following:

[0128] Step S501: Using a multi-layer perceptron network to perform feature mapping on the density coefficient, the complexity coefficient, and the shape change coefficient, inputting the mapping result into a Softmax layer to calculate a weight coefficient, performing a weighted summation of the three coefficients based on the weight coefficient to obtain a scene score, performing one-hot encoding on the scene semantic type, and concatenating the scene score and the encoding result to construct a multimodal feature vector;

[0129] Step S502: Use the multimodal feature vector as the state input of the policy network, construct an action space including modeling algorithm selection and parameter adjustment, use the policy gradient algorithm to optimize the network parameters, design a reward function based on the reconstruction accuracy, select the modeling algorithm by exploring and utilizing the balance strategy, and determine the corresponding parameter combination according to the reward function.

[0130] Optionally, this embodiment proposes a scenario-adaptive modeling method based on deep reinforcement learning, which realizes intelligent selection of modeling algorithms and parameters through multimodal feature analysis and strategy optimization.

[0131] Feature mapping uses a multi-layer perceptron network structure. This embodiment designs a three-layer fully connected network, using ReLU activation functions between each layer to enhance nonlinear expression capabilities. The network performs feature transformations on the density coefficient, complexity coefficient, and shape change coefficient, mapping features of different scales into a unified feature space.

[0132] Softmax normalization is used for weight calculation. This embodiment converts the feature mapping results into a probability distribution through the Softmax layer, ensuring that the sum of the weights of the three coefficients is 1. The weight coefficients reflect the importance of different features in the scene scoring and can adaptively adjust the contribution of each feature.

[0133] Scene scoring integrates multidimensional feature information. This embodiment uses a weighted summation approach to comprehensively consider the scene's geometric density, observation complexity, and shape variation characteristics. The scoring result quantifies the overall modeling difficulty of the scene and provides a basis for subsequent algorithm selection.

[0134] Semantic information is encoded using one-hot encoding. This embodiment converts scene semantic types into binary vectors, with each position corresponding to a semantic category. By concatenating scene scores and semantic encoding, a multimodal feature vector containing both geometric and semantic information is constructed.

[0135] The policy network uses an actor-critic architecture. This embodiment uses a multimodal feature vector as state input, and the network outputs a probability distribution selected by the modeling algorithm and a continuous value of parameter adjustments. The actor network is responsible for generating actions, while the critic network evaluates the state-value function.

[0136] Action space design considers algorithm selection and parameter adjustment. This example uses commonly used 3D reconstruction algorithms (such as TSDF fusion and Poisson reconstruction) as discrete actions and maps key parameters (such as voxel size and smoothing factor) into a continuous action space.

[0137] Policy optimization uses the Proximal Policy Optimization (PPO) algorithm. This embodiment stabilizes the policy update process through importance sampling and trust region constraints. Gradient estimation uses a generalized advantage function to reduce the variance of policy updates.

[0138] The reward function is designed based on reconstruction quality assessment. This embodiment comprehensively considers reconstruction accuracy, completeness, and geometric consistency to construct a multi-objective reward function. Different weight coefficients are set for scene regions of different semantic types to guide the network to learn scene-adaptive modeling strategies.

[0139] In practical applications, this embodiment can automatically select an appropriate modeling method based on scene characteristics. For flat wall areas, a parametric reconstruction algorithm is preferred; for complex object surfaces, a voxel-based fine reconstruction method is selected.

[0140] This embodiment addresses the difficulty of traditional fixed-parameter modeling methods in adapting to complex scenarios. By leveraging deep reinforcement learning, it achieves online optimization of modeling strategies, improving the adaptability and robustness of 3D reconstruction. This solution is valuable in applications such as large-scale scene reconstruction and intelligent modeling, significantly improving reconstruction efficiency and quality.

[0141] In one embodiment of the method for dynamic modeling of scene space three-dimensional model based on multimodal data of the present application, see Figure 6 , and can also include the following:

[0142] Step S601: Input the video sequence and the depth information into a three-dimensional convolutional neural network to extract spatiotemporal features, restore the scene spatial resolution through a deconvolution layer, generate a dense optical flow field, construct a topological adjacency graph for the point cloud data, calculate the displacement difference of point cloud nodes at adjacent time points, and generate a vector field describing the local deformation;

[0143] Step S602: Perform gravity compensation and integration on the acceleration sequence to calculate the displacement, integrate the angular velocity sequence to obtain the rotation quaternion, construct an extended Kalman filter state equation, use the displacement and the rotation quaternion as observation quantities, recursively estimate the six-degree-of-freedom pose parameters of the camera based on the kinematic model, and generate a camera motion trajectory.

[0144] Optionally, this embodiment proposes a scene dynamic feature extraction and camera trajectory estimation method based on multi-source data fusion, which realizes accurate reconstruction and positioning of dynamic scenes through the combination of deep learning and filtering optimization.

[0145] Spatiotemporal feature extraction utilizes a three-dimensional convolutional architecture. This embodiment designs a multi-scale three-dimensional convolutional neural network that simultaneously processes both the spatial and temporal dimensions of a video sequence using 3D convolution kernels. The network comprises multiple residual modules, each of which uses spatially and temporally separated convolution operations to improve computational efficiency for feature extraction.

[0146] Dense optical flow field generation utilizes a multi-layer deconvolution architecture. This embodiment uses skip connections to fuse feature information from different layers, gradually restoring the scene's spatial resolution. Dilated convolutions are introduced during the deconvolution process to expand the receptive field and enhance the network's ability to model large motion displacements.

[0147] Point cloud deformation analysis is based on a graph structure. This example first constructs a K-nearest neighbor topology graph of the point cloud, where each node contains the point's 3D coordinates and a local feature description. By calculating the displacement differences between corresponding nodes at adjacent moments, a vector field describing the local deformation of the scene is generated.

[0148] IMU data preprocessing eliminates the effects of gravity. This embodiment uses attitude estimation to determine the direction of gravity and gravity compensates the acceleration data. After bias correction and integration, displacement information describing camera motion is obtained.

[0149] Rotation parameter calculations are represented using quaternions. This embodiment numerically integrates the angular velocity sequence and updates the camera's rotation state through quaternion multiplication. Quaternion representation avoids the gimbal lock problem associated with Euler angles and ensures the continuity of attitude estimation.

[0150] The extended Kalman filter design considers multi-source observations. This implementation uses camera position, velocity, attitude, and IMU bias as state vectors to construct a kinematic model that incorporates IMU pre-integration. The observation equations incorporate visual reprojection errors and IMU measurement constraints to achieve robust state estimation.

[0151] The recursive estimation process uses an adaptive update strategy. This embodiment dynamically adjusts the Kalman gain based on the statistical characteristics of the observation noise, balancing the credibility of the prediction model and the observation data. When large movements or rapid turns occur, the weight of the IMU observations is increased to improve tracking stability.

[0152] In dynamic scene reconstruction applications, this embodiment accurately captures moving objects within the scene. For dynamic elements such as human motion and object deformation, optical flow and deformation vector fields provide precise motion descriptions. Furthermore, stable camera trajectory estimation provides a reliable perspective reference for dynamic reconstruction.

[0153] This embodiment addresses the adaptability issues of traditional static reconstruction methods in dynamic scenes. By integrating multi-source data fusion and deep learning methods, the accuracy and robustness of dynamic scene understanding are improved. This solution is valuable in applications requiring real-time dynamic reconstruction, such as augmented reality and robotic navigation, and can provide accurate dynamic features and camera pose information for subsequent scene reconstruction and interaction.

[0154] In one embodiment of the method for dynamic modeling of scene space three-dimensional model based on multimodal data of the present application, see Figure 7 , and can also include the following:

[0155] Step S701: Construct a graph convolutional network to perform feature aggregation on the three input fields, capture temporal variation features through a gated recurrent unit, use a graph pooling layer to output a dynamic region probability map, determine the dynamic region boundary based on the probability map, perform quadtree meshing within the dynamic region, and use the modeling algorithm to geometrically reconstruct the mesh nodes;

[0156] Step S702: Setting a baseline threshold according to the scene semantic type, dynamically adjusting the baseline threshold based on the scene complexity, calculating the geometric error between the reconstructed point cloud and the original point cloud of the dynamic area, comparing the geometric error with the dynamic threshold, and triggering the policy network to re-execute the modeling algorithm selection.

[0157] Optionally, this embodiment first processes the input optical flow field, deformation vector field, and probability field using a graph convolutional network. During feature aggregation, each node exchanges information with its neighboring nodes through a message passing mechanism, achieving a gradual fusion of features from local to global. Through multi-layer graph convolution operations, the network can effectively capture spatial correlations in the scene.

[0158] This example builds on feature extraction by introducing a recurrent neural network structure to analyze dynamic scene changes. Through forward and backward bidirectional processing, the network fully leverages information from both past and future frames, accurately capturing temporal dependencies within the scene. A selective update mechanism ensures that important dynamic information is preserved.

[0159] This example uses a hierarchical feature clustering approach to generate a dynamic region probability map. The network gradually reduces the spatial resolution of the feature map during pooling, while preserving significant dynamic features. Based on the generated probability map, a region growing algorithm is used to determine the boundaries of the dynamic region.

[0160] This embodiment performs adaptive meshing within a defined dynamic region. The meshing process uses a quadtree structure, recursively subdividing the mesh based on the geometric characteristics of the local region. In areas with complex boundaries or rich details, the meshing is refined to ensure reconstruction accuracy.

[0161] This embodiment applies the selected modeling algorithm to mesh nodes for geometric reconstruction. The reconstruction process integrates depth information and point cloud data, employing appropriate reconstruction strategies for different surface types. For example, plane fitting is used for flat areas, while implicit function reconstruction is employed for complex curved surfaces.

[0162] This embodiment sets a baseline threshold for reconstruction quality assessment based on the scene's semantic type. Different semantic types have different geometric features and reconstruction requirements, and therefore require differentiated quality assessment criteria. For example, wall areas require higher flatness, while object surfaces prioritize detail preservation.

[0163] This embodiment introduces a scene complexity adaptive mechanism to dynamically adjust the threshold. Complexity assessment takes into account factors such as local curvature changes and depth discontinuities. In areas of high complexity, the threshold tolerance is increased to avoid over-refinement of the reconstruction.

[0164] This embodiment performs a multi-dimensional quality assessment on the reconstruction results. Evaluation metrics include geometric distances between point clouds and normal vector consistency. When the evaluation result exceeds a dynamic threshold, the strategy network is triggered to reselect the algorithm and optimize the parameters.

[0165] In indoor scene reconstruction applications, this embodiment can accurately identify and reconstruct dynamic objects. For example, the system can capture the motion characteristics of moving human bodies, opening and closing doors and windows, and accurately reconstruct them. Meanwhile, it maintains stable reconstruction quality for static architectural structures.

[0166] This embodiment addresses the difficulty traditional reconstruction methods have in adapting to dynamic scenes. Through dynamic feature analysis and adaptive quality control, the accuracy and robustness of scene reconstruction are significantly improved. This technical solution has significant value in applications such as real-time 3D reconstruction and augmented reality, providing users with high-quality dynamic scene reconstruction services.

[0167] In order to effectively improve the quality and efficiency of 3D reconstruction of large-scale complex scenes, the present application provides an embodiment of a scene space 3D model dynamic modeling device based on multimodal data for realizing all or part of the content of the scene space 3D model dynamic modeling method based on multimodal data, see Figure 8 The scene space three-dimensional model dynamic modeling device based on multimodal data specifically includes the following contents:

[0168] A multi-source data processing module 10 is configured to collect video sequences, point cloud data, and IMU data of large-scale indoor and outdoor scenes, perform deep learning noise reduction and distortion correction on the video sequences, perform spatial downsampling and denoising on the point cloud data, perform zero-bias calibration and calibration on the IMU data, extract ORB image features and depth information from the video sequences, extract normal vector features and edge features from the point cloud data, extract acceleration sequences and angular velocity sequences from the IMU data, establish spatiotemporal correspondences among the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequences, and the angular velocity sequences, and fuse the correspondences using a particle filter algorithm to obtain multi-source scene information.

[0169] A scene space processing module 20 is configured to construct a Transformer-based multimodal scene analysis network, perform instance segmentation on the multi-source scene information to obtain a scene semantic type, perform octree partitioning on the point cloud data to calculate a spatial density coefficient, use a frustum projection method to statistically calculate a depth occlusion map of the multi-source scene information to obtain a scene complexity coefficient, construct a geometric descriptor based on the normal vector features and the edge features to calculate a shape variation coefficient, perform nonlinear weighting on the density coefficient, the complexity coefficient, and the shape variation coefficient to obtain a scene score, construct a multimodal feature representation based on the scene semantic type and the scene score, and select a modeling algorithm and parameter combination in a policy network trained by deep reinforcement learning;

[0170] The dynamic modeling module 30 is used to use a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculate the topological structure change of the point cloud data to generate a deformation vector field, estimate the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence, input the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, apply the modeling algorithm and the parameter combination to the dynamic area for adaptive mesh subdivision and geometric constraint reconstruction, calculate a dynamic threshold according to the semantic type of the scene, and reselect the modeling algorithm in the strategy network to reconstruct the dynamic area when the reconstruction error of the dynamic area exceeds the dynamic threshold.

[0171] From the above description, it can be seen that the dynamic modeling device of the scene space three-dimensional model based on multimodal data provided in the embodiment of the present application can establish a spatiotemporal correspondence by fusing the feature information of video sequences, point cloud data and IMU data using a particle filter algorithm. The Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates scene scores based on spatial density, depth occlusion and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0172] From a hardware perspective, in order to effectively improve the quality and efficiency of 3D reconstruction of large-scale complex scenes, this application provides an embodiment of an electronic device for implementing all or part of the method for dynamic modeling of a scene space 3D model based on multimodal data. The electronic device specifically includes the following:

[0173] A processor, a memory, a communications interface, and a bus; wherein the processor, the memory, and the communications interface communicate with each other via the bus; the communications interface is used to implement information transmission between a dynamic modeling device for a scene space three-dimensional model based on multimodal data and related devices such as a core business system, a user terminal, and related databases; the logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., but this embodiment is not limited thereto. In this embodiment, the logic controller can be implemented with reference to the embodiment of the method for dynamic modeling a scene space three-dimensional model based on multimodal data and the embodiment of the device for dynamic modeling a scene space three-dimensional model based on multimodal data in the embodiment, the contents of which are incorporated herein, and repeated parts are not repeated.

[0174] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0175] In practical applications, part of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data can be executed on the electronic device side as described above, or all operations can be completed on the client device. The specific selection can be based on the processing capabilities of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed on the client device, the client device may also include a processor.

[0176] The aforementioned client device may include a communication module (i.e., a communication unit) capable of establishing a communication connection with a remote server to facilitate data transmission with the server. The server may include a server at the task scheduling center or, in other implementation scenarios, a server on an intermediate platform, such as a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may comprise a single computer device, a server cluster consisting of multiple servers, or a distributed server configuration.

[0177] Figure 9 Schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Figure 9 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that the Figure 9is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0178] In one embodiment, the function of the method for dynamic modeling of a scene space 3D model based on multimodal data can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0179] Step S101: collecting video sequences, point cloud data, and IMU data of large-scale indoor and outdoor scenes, performing deep learning noise reduction and distortion correction on the video sequences, performing spatial downsampling and denoising on the point cloud data, performing zero-bias calibration and calibration on the IMU data, extracting ORB image features and depth information from the video sequences, extracting normal vector features and edge features from the point cloud data, extracting acceleration sequences and angular velocity sequences from the IMU data, establishing spatiotemporal correspondences among the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequences, and the angular velocity sequences, and fusing the correspondences using a particle filter algorithm to obtain multi-source scene information;

[0180] Step S102: constructing a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain a scene semantic type, performing octree partitioning on the point cloud data to calculate a spatial density coefficient, using a frustum projection method to statistically calculate a depth occlusion map of the multi-source scene information to obtain a scene complexity coefficient, constructing a geometric descriptor based on the normal vector features and the edge features to calculate a shape variation coefficient, performing nonlinear weighting on the density coefficient, the complexity coefficient, and the shape variation coefficient to obtain a scene score, constructing a multimodal feature representation based on the scene semantic type and the scene score, and selecting a modeling algorithm and parameter combination in a policy network trained by deep reinforcement learning;

[0181] Step S103: Using a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculating the topological structure change of the point cloud data to generate a deformation vector field, estimating the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence, inputting the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, applying the modeling algorithm and the parameter combination to the dynamic area to perform adaptive mesh subdivision and geometric constraint reconstruction, calculating a dynamic threshold according to the semantic type of the scene, and reselecting the modeling algorithm in the strategy network to reconstruct the dynamic area when the reconstruction error of the dynamic area exceeds the dynamic threshold.

[0182] As can be seen from the above description, the electronic device provided in the embodiment of the present application establishes a spatiotemporal correspondence by fusing the feature information of video sequences, point cloud data, and IMU data, and adopts a particle filter algorithm. The Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates scene scores by combining spatial density, depth occlusion, and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0183] In another embodiment, the scene space three-dimensional model dynamic modeling device based on multimodal data can be configured separately from the central processing unit 9100. For example, the scene space three-dimensional model dynamic modeling device based on multimodal data can be configured as a chip connected to the central processing unit 9100, and the function of the scene space three-dimensional model dynamic modeling method based on multimodal data can be realized through the control of the central processing unit.

[0184] like Figure 9 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Figure 9 In addition, the electronic device 9600 may also include all components shown in Figure 9 For components not shown, reference may be made to the prior art.

[0185] like Figure 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0186] Memory 9140 can be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It can store the aforementioned failure-related information and also store programs that execute the relevant information. The CPU 9100 can execute the programs stored in memory 9140 to implement information storage or processing.

[0187] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 may be, for example, a keypad or touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0188] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), or SIM card. Alternatively, it may be a memory that retains information even when power is off, can be selectively erased, and is capable of storing additional data. Examples of such memory are sometimes referred to as EPROMs. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs, or processes used by the central processing unit 9100 to execute operations of the electronic device 9600.

[0189] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, images, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various driver programs for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0190] The communication module 9110 is a transmitter / receiver that transmits and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0191] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless local area network modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130, providing audio output via the speaker 9131 and receiving audio input from the microphone 9132, thereby implementing common telecommunication functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor 9130 is coupled to the central processing unit 9100, enabling local recording via the microphone 9132 and playback of stored audio via the speaker 9131.

[0192] Embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the method for dynamically modeling a scene space three-dimensional model based on multimodal data, in which the execution subject is a server or a client in the above-mentioned embodiment. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all steps of the method for dynamically modeling a scene space three-dimensional model based on multimodal data, in which the execution subject is a server or a client in the above-mentioned embodiment, are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0193] Step S101: collecting video sequences, point cloud data, and IMU data of large-scale indoor and outdoor scenes, performing deep learning noise reduction and distortion correction on the video sequences, performing spatial downsampling and denoising on the point cloud data, performing zero-bias calibration and calibration on the IMU data, extracting ORB image features and depth information from the video sequences, extracting normal vector features and edge features from the point cloud data, extracting acceleration sequences and angular velocity sequences from the IMU data, establishing spatiotemporal correspondences among the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequences, and the angular velocity sequences, and fusing the correspondences using a particle filter algorithm to obtain multi-source scene information;

[0194] Step S102: constructing a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain a scene semantic type, performing octree partitioning on the point cloud data to calculate a spatial density coefficient, using a frustum projection method to statistically calculate a depth occlusion map of the multi-source scene information to obtain a scene complexity coefficient, constructing a geometric descriptor based on the normal vector features and the edge features to calculate a shape variation coefficient, performing nonlinear weighting on the density coefficient, the complexity coefficient, and the shape variation coefficient to obtain a scene score, constructing a multimodal feature representation based on the scene semantic type and the scene score, and selecting a modeling algorithm and parameter combination in a policy network trained by deep reinforcement learning;

[0195] Step S103: Using a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculating the topological structure change of the point cloud data to generate a deformation vector field, estimating the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence, inputting the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, applying the modeling algorithm and the parameter combination to the dynamic area to perform adaptive mesh subdivision and geometric constraint reconstruction, calculating a dynamic threshold according to the semantic type of the scene, and reselecting the modeling algorithm in the strategy network to reconstruct the dynamic area when the reconstruction error of the dynamic area exceeds the dynamic threshold.

[0196] As can be seen from the above description, the computer-readable storage medium provided in the embodiment of the present application establishes a spatiotemporal correspondence by fusing the feature information of video sequences, point cloud data, and IMU data, and adopts a particle filter algorithm. The Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates scene scores by combining spatial density, depth occlusion, and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0197] The embodiments of the present application also provide a computer program product capable of implementing all steps of the method for dynamically modeling a scene space three-dimensional model based on multimodal data in the above-mentioned embodiment, where the execution subject is a server or a client. When the computer program / instructions are executed by a processor, the computer program / instructions implement the steps of the method for dynamically modeling a scene space three-dimensional model based on multimodal data. For example, the computer program / instructions implement the following steps:

[0198] Step S101: collecting video sequences, point cloud data, and IMU data of large-scale indoor and outdoor scenes, performing deep learning noise reduction and distortion correction on the video sequences, performing spatial downsampling and denoising on the point cloud data, performing zero-bias calibration and calibration on the IMU data, extracting ORB image features and depth information from the video sequences, extracting normal vector features and edge features from the point cloud data, extracting acceleration sequences and angular velocity sequences from the IMU data, establishing spatiotemporal correspondences among the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequences, and the angular velocity sequences, and fusing the correspondences using a particle filter algorithm to obtain multi-source scene information;

[0199] Step S102: constructing a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain a scene semantic type, performing octree partitioning on the point cloud data to calculate a spatial density coefficient, using a frustum projection method to statistically calculate a depth occlusion map of the multi-source scene information to obtain a scene complexity coefficient, constructing a geometric descriptor based on the normal vector features and the edge features to calculate a shape variation coefficient, performing nonlinear weighting on the density coefficient, the complexity coefficient, and the shape variation coefficient to obtain a scene score, constructing a multimodal feature representation based on the scene semantic type and the scene score, and selecting a modeling algorithm and parameter combination in a policy network trained by deep reinforcement learning;

[0200] Step S103: Using a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculating the topological structure change of the point cloud data to generate a deformation vector field, estimating the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence, inputting the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, applying the modeling algorithm and the parameter combination to the dynamic area to perform adaptive mesh subdivision and geometric constraint reconstruction, calculating a dynamic threshold according to the semantic type of the scene, and reselecting the modeling algorithm in the strategy network to reconstruct the dynamic area when the reconstruction error of the dynamic area exceeds the dynamic threshold.

[0201] As can be seen from the above description, the computer program product provided in the embodiment of the present application establishes a spatiotemporal correspondence by fusing the feature information of video sequences, point cloud data, and IMU data, and adopts a particle filter algorithm. The Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates scene scores by combining spatial density, depth occlusion, and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0202] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatuses, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0203] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0206] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A method for dynamic modeling of a scene space three-dimensional model based on multimodal data, characterized in that: The method comprises: Collect video sequences, point cloud data, and IMU data of large-scale indoor and outdoor scenes, perform deep learning noise reduction and distortion correction on the video sequences, perform spatial downsampling and denoising on the point cloud data, perform zero-bias calibration and calibration on the IMU data, extract ORB image features and depth information from the video sequences, extract normal vector features and edge features from the point cloud data, extract acceleration sequences and angular velocity sequences from the IMU data, establish spatiotemporal correspondences among the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence, and the angular velocity sequence, and use a particle filter algorithm to fuse these correspondences to obtain multi-source scene information; Mapping the multi-source scene information into a feature sequence, positionally encoding the feature sequence, calculating the association weights between features through a multi-head self-attention mechanism, transforming the weighted features using a feedforward neural network, constructing a semantic segmentation head network to classify scene elements, generating a scene semantic type label map, recursively partitioning the point cloud space into an octree, counting the number of point clouds in each leaf node to calculate the density coefficient, constructing a camera frustum model to project the scene, calculating the projection overlapping area to obtain the complexity coefficient, combining the normal vector feature and the edge feature to construct a local geometric descriptor, calculating the variance of the descriptor to obtain a shape variation coefficient, nonlinearly weighting the density coefficient, the complexity coefficient, and the shape variation coefficient to obtain a scene score, constructing a multimodal feature representation based on the scene semantic type and the scene score, and selecting a modeling algorithm and parameter combination in a policy network trained by deep reinforcement learning; The video sequence and the depth information are input into a three-dimensional convolutional neural network to extract spatiotemporal features, the spatial resolution of the scene is restored through a deconvolution layer, a dense optical flow field is generated, a topological adjacency graph is constructed for the point cloud data, the displacement differences of the point cloud nodes at adjacent moments are calculated, and a vector field describing the local deformation is generated; gravity compensation and integration are performed on the acceleration sequence to calculate the displacement, the angular velocity sequence is integrated to obtain the rotation quaternion, an extended Kalman filter state equation is constructed, the displacement and the rotation quaternion are used as observation quantities, the six-degree-of-freedom pose parameters of the camera are recursively estimated based on the kinematic model, and the camera motion trajectory is generated; the three-dimensional motion field, the deformation vector field and the motion trajectory are input into a graph neural network to predict the dynamic area of ​​the scene, the modeling algorithm and the parameter combination are applied to the dynamic area for adaptive mesh subdivision and geometric constraint reconstruction, the dynamic threshold is calculated according to the semantic type of the scene, and when the reconstruction error of the dynamic area exceeds the dynamic threshold, the modeling algorithm is reselected in the policy network to reconstruct the dynamic area.

2. The method for dynamic modeling of a scene space three-dimensional model based on multimodal data according to claim 1, characterized in that: The method includes inputting the three-dimensional motion field, the deformation vector field, and the motion trajectory into a graph neural network to predict a dynamic region of a scene, applying the modeling algorithm and the parameter combination to the dynamic region to perform adaptive mesh subdivision and geometric constraint reconstruction, calculating a dynamic threshold according to the scene semantic type, and reselecting a modeling algorithm in the strategy network to reconstruct the dynamic region when a reconstruction error of the dynamic region exceeds the dynamic threshold, including: Constructing a graph convolutional network to aggregate features of the three input fields, capturing temporal variation features through a gated recurrent unit, and using a graph pooling layer to output a probability map of the dynamic region. Based on the probability map, the boundaries of the dynamic region are determined, quadtree meshing is performed within the dynamic region, and the mesh nodes are geometrically reconstructed using the modeling algorithm. A baseline threshold is set according to the scene semantic type, the baseline threshold is dynamically adjusted based on the scene complexity, the geometric error between the reconstructed point cloud and the original point cloud of the dynamic area is calculated, the geometric error is compared with the dynamic threshold, and the policy network is triggered to re-execute the modeling algorithm selection.

3. A dynamic modeling device for scene space three-dimensional model based on multimodal data, characterized in that: The device comprises: A multi-source data processing module is used to collect video sequences, point cloud data, and IMU data of large-scale indoor and outdoor scenes, perform deep learning noise reduction and distortion correction on the video sequences, perform spatial downsampling and denoising on the point cloud data, perform zero-bias calibration and calibration on the IMU data, extract ORB image features and depth information from the video sequences, extract normal vector features and edge features from the point cloud data, extract acceleration sequences and angular velocity sequences from the IMU data, establish spatiotemporal correspondences among the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence, and the angular velocity sequence, and fuse the correspondences using a particle filter algorithm to obtain multi-source scene information; A scene space processing module is used to map the multi-source scene information into a feature sequence, perform position encoding on the feature sequence, calculate the association weights between features through a multi-head self-attention mechanism, transform the weighted features using a feedforward neural network, construct a semantic segmentation head network to classify scene elements, generate a scene semantic type label map, recursively partition the point cloud space into an octree, count the number of point clouds in each leaf node to calculate the density coefficient, construct a camera frustum model to project the scene, calculate the projection overlapping area to obtain the complexity coefficient, combine the normal vector feature and the edge feature to construct a local geometric descriptor, calculate the variance of the descriptor to obtain a shape variation coefficient, nonlinearly weight the density coefficient, the complexity coefficient and the shape variation coefficient to obtain a scene score, construct a multimodal feature representation based on the scene semantic type and the scene score, and select a modeling algorithm and parameter combination in a policy network trained by deep reinforcement learning; A dynamic modeling module is used to input the video sequence and the depth information into a three-dimensional convolutional neural network to extract spatiotemporal features, restore the scene spatial resolution through a deconvolution layer, generate a dense optical flow field, construct a topological adjacency graph for the point cloud data, calculate the displacement differences of point cloud nodes at adjacent moments, and generate a vector field describing local deformation; perform gravity compensation and integral calculation of the acceleration sequence, integrate the angular velocity sequence to obtain a rotation quaternion, construct an extended Kalman filter state equation, use the displacement and the rotation quaternion as observation quantities, recursively estimate the six-degree-of-freedom pose parameters of the camera based on a kinematic model, generate a camera motion trajectory, input the three-dimensional motion field, the deformation vector field, and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, apply the modeling algorithm and the parameter combination to the dynamic area for adaptive mesh subdivision and geometric constraint reconstruction, calculate a dynamic threshold according to the scene semantic type, and reselect the modeling algorithm in the strategy network to reconstruct the dynamic area when the reconstruction error of the dynamic area exceeds the dynamic threshold.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data as described in any one of claims 1 to 2 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data as described in any one of claims 1 to 2 are implemented.

Images