Scene space three-dimensional model dynamic modeling method based on multi-modal data

Abstract

The embodiment of the invention provides a scene space three-dimensional model dynamic modeling method based on multi-modal data, and the method comprises the steps: fusing the feature information of a video sequence, point cloud data and IMU data, and building a space-time corresponding relation through employing a particle filtering algorithm. Scene semantic understanding is realized based on a Transform multi-modal scene analysis network, scene scores are calculated in combination with spatial density, depth occlusion and shape change features, and an optimal modeling algorithm is selected through a strategy network of deep reinforcement learning training. Meanwhile, scene dynamic features are analyzed by using a deep neural network and a graph neural network, adaptive grid subdivision and geometric constraint reconstruction of a dynamic region are realized, and a reconstruction threshold is dynamically adjusted according to a scene semantic type. According to the method, the three-dimensional reconstruction quality and efficiency of a large-scale complex scene are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a method for dynamic modeling of a scene space three-dimensional model based on multimodal data. Background Art

[0002] Traditional scene 3D reconstruction methods face many challenges when dealing with large-scale complex scenes. Existing single data source modeling solutions are difficult to deal with complex situations such as occlusion, lighting changes, and dynamic targets, while simple multi-source data fusion methods cannot fully utilize the complementary advantages of various sensor data. Although the introduction of deep learning technology has improved modeling performance, existing systems still have problems such as poor model adaptability and low reconstruction efficiency.

[0003] Especially when dealing with large-scale complex indoor and outdoor scenes, traditional methods have difficulty accurately identifying and processing dynamic areas, and cannot effectively balance reconstruction accuracy and computational efficiency. At the same time, fixed modeling strategies cannot be dynamically adjusted according to scene characteristics, resulting in significant differences in reconstruction quality under different types of scenes. In addition, existing systems lack in-depth understanding and utilization of scene semantic information, making it difficult to achieve intelligent modeling parameter configuration.

[0004] The key issues that need to be solved urgently include: how to achieve deep fusion of multimodal data, establish accurate representation of scene features, build an intelligent modeling strategy selection mechanism, and improve the system's adaptability to dynamic scenes. This is not only related to the quality and efficiency of 3D reconstruction, but also an important basis for realizing intelligent scene understanding and modeling. Summary of the invention

[0005] In response to the problems in the prior art, the present application provides a method for dynamic modeling of scene space three-dimensional models based on multimodal data, which can effectively improve the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0006] In order to solve at least one of the above problems, the present application provides the following technical solutions: In a first aspect, the present application provides a method for dynamic modeling of a scene space three-dimensional model based on multimodal data, comprising: Collect video sequences, point cloud data and IMU data of large-scale indoor and outdoor scenes, perform deep learning noise reduction and distortion correction on the video sequences, perform spatial downsampling and denoising on the point cloud data, perform zero bias calibration and calibration on the IMU data, extract ORB image features and depth information from the video sequences, extract normal vector features and edge features from the point cloud data, extract acceleration sequences and angular velocity sequences from the IMU data, establish a spatiotemporal correspondence between the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence and the angular velocity sequence, and use a particle filter algorithm to fuse the correspondence to obtain multi-source scene information; Construct a Transformer-based multimodal scene analysis network, perform instance segmentation on the multi-source scene information to obtain the scene semantic type, perform octree partitioning on the point cloud data to calculate the spatial density coefficient, use the cone projection method to statistically calculate the depth occlusion map of the multi-source scene information to obtain the scene complexity coefficient, construct a geometric descriptor based on the normal vector feature and the edge feature to calculate the shape change coefficient, perform nonlinear weighting on the density coefficient, the complexity coefficient and the shape change coefficient to obtain a scene score, construct a multimodal feature representation according to the scene semantic type and the scene score, and select a modeling algorithm and parameter combination in the policy network trained by deep reinforcement learning; The video sequence and the depth information are fused by using a deep neural network to obtain a three-dimensional motion field of the scene, the topological structure change of the point cloud data is calculated to generate a deformation vector field, the camera motion trajectory is estimated based on the fusion of the acceleration sequence and the angular velocity sequence, the three-dimensional motion field, the deformation vector field and the motion trajectory are input into a graph neural network to predict the dynamic area of ​​the scene, the modeling algorithm and the parameter combination are applied to the dynamic area to perform adaptive mesh subdivision and geometric constraint reconstruction, the dynamic threshold is calculated according to the scene semantic type, and when the reconstruction error of the dynamic area exceeds the dynamic threshold, the modeling algorithm is reselected in the strategy network to reconstruct the dynamic area.

[0007] Furthermore, the performing deep learning noise reduction and distortion correction on the video sequence, performing spatial downsampling and denoising on the point cloud data, performing zero bias calibration and calibration on the IMU data, extracting ORB image features and depth information from the video sequence, extracting normal vector features and edge features from the point cloud data, and extracting acceleration sequences and angular velocity sequences from the IMU data, includes: The video sequence is input into a U-shaped convolutional neural network for multi-scale feature extraction, the feature map is subjected to skip connection fusion to obtain a noise distribution map, the denoised video sequence is reconstructed based on the noise distribution map, the checkerboard corner point positions in the video sequence are calculated, a nonlinear equation group based on the Zhang camera model is constructed, the Levenberg-Marquardt algorithm is used to solve the equation group to obtain lens distortion parameters, and the denoised video sequence is corrected according to the distortion parameters; Perform FAST corner detection on the video sequence to obtain a feature point set, calculate the BRIEF descriptor of the feature point set to construct ORB features, use a binocular disparity algorithm to calculate a depth map, construct a K nearest neighbor graph for the point cloud data, calculate the covariance matrix of each point based on the K nearest neighbor graph to obtain a normal vector, use an edge detection operator to extract point cloud edge features, perform bandpass filtering on the IMU data to obtain an acceleration sequence, and perform gyroscope integration on the IMU data to obtain an angular velocity sequence.

[0008] Furthermore, the establishing of the spatiotemporal correspondence between the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence and the angular velocity sequence, and fusing the correspondence using a particle filter algorithm to obtain multi-source scene information includes: Construct a KD tree index structure for the ORB image features, calculate feature matching pairs based on nearest neighbor search, remove abnormal matching points using the RANSAC algorithm, project the depth information into three-dimensional space to establish a point cloud mapping relationship, perform spatial position indexing on the normal vector features and the edge features, establish a feature association table aligned with timestamps, and construct a feature spatiotemporal consistency verification model using camera pose and IMU motion constraints; A state space equation based on the feature spatiotemporal consistency verification model is constructed, the particle swarm state vector and weight are initialized, the likelihood function between the observed data in the feature association table and the particle state is calculated, the particle weight is updated according to the likelihood function, a resampling process is performed to eliminate weight degradation, and the fused multi-source scene information is obtained based on the weighted average value of the particle swarm state vector.

[0009] Furthermore, the constructing of a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain scene semantic types, performing octree partitioning on the point cloud data to calculate a spatial density coefficient, using a frustum projection method to statistically calculate a depth occlusion map of the multi-source scene information to obtain a scene complexity coefficient, and constructing a geometric descriptor based on the normal vector feature and the edge feature to calculate a shape variation coefficient, includes: Mapping the multi-source scene information into a feature sequence, encoding the feature sequence by position, calculating the association weights between features through a multi-head self-attention mechanism, transforming the weighted features using a feedforward neural network, constructing a semantic segmentation head network to classify scene elements, and generating a scene semantic type label map; The point cloud space is recursively divided into octrees, the number of point clouds in each leaf node is counted to calculate the density coefficient, a camera frustum model is constructed to project the scene, the projection overlapping area is calculated to obtain the complexity coefficient, the normal vector feature and the edge feature are combined to construct a local geometric descriptor, and the variance of the descriptor is calculated to obtain the shape change coefficient.

[0010] Furthermore, the density coefficient, the complexity coefficient and the shape change coefficient are nonlinearly weighted to obtain a scene score, a multimodal feature representation is constructed according to the scene semantic type and the scene score, and a modeling algorithm and a parameter combination are selected in a policy network trained by deep reinforcement learning, including: Using a multi-layer perceptron network to perform feature mapping on the density coefficient, the complexity coefficient and the shape change coefficient, inputting the mapping result into a Softmax layer to calculate a weight coefficient, performing weighted summation on the three coefficients based on the weight coefficient to obtain a scene score, performing one-hot encoding on the scene semantic type, and concatenating the scene score with the encoding result to construct a multimodal feature vector; The multimodal feature vector is used as the state input of the policy network, an action space including modeling algorithm selection and parameter adjustment is constructed, the policy gradient algorithm is used to optimize the network parameters, a reward function is designed based on the reconstruction accuracy, the modeling algorithm is selected by exploring and utilizing the balance strategy, and the corresponding parameter combination is determined according to the reward function.

[0011] Furthermore, the method of using a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculating the topological structure change of the point cloud data to generate a deformation vector field, and estimating the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence includes: Input the video sequence and the depth information into a three-dimensional convolutional neural network to extract spatiotemporal features, restore the scene spatial resolution through a deconvolution layer, generate a dense optical flow field, construct a topological adjacency graph for the point cloud data, calculate the displacement difference of the point cloud nodes at adjacent times, and generate a vector field describing the local deformation; The acceleration sequence is gravity compensated and integrated to calculate the displacement, the angular velocity sequence is integrated to obtain the rotation quaternion, an extended Kalman filter state equation is constructed, the displacement and the rotation quaternion are used as observation quantities, the six-degree-of-freedom posture parameters of the camera are recursively estimated based on the kinematic model, and the camera motion trajectory is generated.

[0012] Furthermore, the three-dimensional motion field, the deformation vector field and the motion trajectory are input into a graph neural network to predict a scene dynamic area, the modeling algorithm and the parameter combination are applied to the dynamic area for adaptive mesh subdivision and geometric constraint reconstruction, a dynamic threshold is calculated according to the scene semantic type, and when the reconstruction error of the dynamic area exceeds the dynamic threshold, a modeling algorithm is reselected in the strategy network to reconstruct the dynamic area, including: Construct a graph convolutional network to aggregate the features of the three input fields, capture the temporal variation features through the gated recurrent unit, use the graph pooling layer to output the dynamic region probability map, determine the dynamic region boundary based on the probability map, perform quadtree meshing within the dynamic region, and use the modeling algorithm to geometrically reconstruct the mesh nodes; A benchmark threshold is set according to the scene semantic type, the benchmark threshold is dynamically adjusted based on the scene complexity, a geometric error between the reconstructed point cloud and the original point cloud of the dynamic area is calculated, the geometric error is compared with the dynamic threshold, and the policy network is triggered to re-execute the modeling algorithm selection.

[0013] In a second aspect, the present application provides a scene space three-dimensional model dynamic modeling device based on multimodal data, comprising: A multi-source data processing module is used to collect video sequences, point cloud data and IMU data of large-scale indoor and outdoor scenes, perform deep learning noise reduction and distortion correction on the video sequence, perform spatial downsampling and denoising on the point cloud data, perform zero bias calibration and calibration on the IMU data, extract ORB image features and depth information from the video sequence, extract normal vector features and edge features from the point cloud data, extract acceleration sequence and angular velocity sequence from the IMU data, establish a spatiotemporal correspondence between the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence and the angular velocity sequence, and use a particle filter algorithm to fuse the correspondence to obtain multi-source scene information; A scene space processing module, for constructing a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain a scene semantic type, performing octree partitioning on the point cloud data to calculate a spatial density coefficient, using a frustum projection method to statistically calculate a depth occlusion map of the multi-source scene information to obtain a scene complexity coefficient, constructing a geometric descriptor based on the normal vector feature and the edge feature to calculate a shape variation coefficient, performing nonlinear weighting on the density coefficient, the complexity coefficient and the shape variation coefficient to obtain a scene score, constructing a multimodal feature representation according to the scene semantic type and the scene score, and selecting a modeling algorithm and a parameter combination in a policy network trained by deep reinforcement learning; A dynamic modeling module is used to use a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculate the topological structure change of the point cloud data to generate a deformation vector field, estimate the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence, input the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, apply the modeling algorithm and the parameter combination to the dynamic area for adaptive mesh subdivision and geometric constraint reconstruction, calculate a dynamic threshold according to the scene semantic type, and when the reconstruction error of the dynamic area exceeds the dynamic threshold, reselect the modeling algorithm in the strategy network to reconstruct the dynamic area.

[0014] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data are implemented.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data.

[0016] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data.

[0017] It can be seen from the above technical solution that the present application provides a method for dynamic modeling of scene space three-dimensional models based on multimodal data, which establishes the space-time correspondence relationship by fusing the feature information of video sequences, point cloud data and IMU data, and adopts a particle filter algorithm. The Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates the scene score by combining spatial density, depth occlusion and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is one of the flow charts of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application; Figure 2 This is a second flow chart of a method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application; Figure 3 The third flowchart of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application; Figure 4 This is a fourth flow chart of a method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application; Figure 5 This is a fifth flow chart of a method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application; Figure 6 This is a sixth flow chart of a method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application; Figure 7 FIG7 is a flow chart of a method for dynamic modeling of a scene space three-dimensional model based on multimodal data in an embodiment of the present application; Figure 8 It is a structural diagram of a scene space three-dimensional model dynamic modeling device based on multimodal data in an embodiment of the present application; Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0020] Reference numerals: Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0023] Taking into account the problems existing in the prior art, the present application provides a method for dynamic modeling of scene space three-dimensional models based on multimodal data, which uses a particle filter algorithm to establish a spatiotemporal correspondence by fusing the feature information of video sequences, point cloud data, and IMU data. The Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates scene scores based on spatial density, depth occlusion, and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0024] In order to effectively improve the quality and efficiency of 3D reconstruction of large-scale complex scenes, the present application provides an embodiment of a method for dynamic modeling of a scene space 3D model based on multimodal data, see Figure 1 The method for dynamic modeling of scene space three-dimensional model based on multimodal data specifically includes the following contents: Step S101: collecting video sequences, point cloud data and IMU data of large-scale indoor and outdoor scenes, performing deep learning noise reduction and distortion correction on the video sequences, performing spatial downsampling and denoising on the point cloud data, performing zero bias calibration and calibration on the IMU data, extracting ORB image features and depth information from the video sequences, extracting normal vector features and edge features from the point cloud data, extracting acceleration sequences and angular velocity sequences from the IMU data, establishing a spatiotemporal correspondence between the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence and the angular velocity sequence, and using a particle filter algorithm to fuse the correspondence to obtain multi-source scene information; Optionally, this embodiment proposes a multi-source data acquisition and preprocessing method, which achieves high-quality acquisition and feature extraction of scene information by fusing video, point cloud and IMU data.

[0025] This embodiment first uses professional-grade camera equipment to collect high-resolution video sequences, obtains high-precision point cloud data through laser radar, and uses IMU sensors to record motion parameters. During the data collection process, the time synchronization and spatial alignment of the three devices are ensured to establish a unified data collection reference system.

[0026] For the processing of video sequences, this embodiment uses an improved U-Net network structure for noise reduction. The network accurately identifies the image noise distribution through multi-level feature extraction and fusion, and realizes adaptive noise reduction. In the distortion correction link, an improved Zhang calibration method is used, a nonlinear equation group is established through chessboard corner point detection, and the LM algorithm is used to solve the camera intrinsic parameters and distortion coefficients.

[0027] The preprocessing of point cloud data adopts an adaptive voxel filtering method. This embodiment dynamically adjusts the voxel size according to the point cloud density distribution, and achieves data downsampling while maintaining geometric details. Through statistical outlier analysis and local surface fitting, noise points and outliers are effectively removed.

[0028] The processing of IMU data focuses on solving the problem of zero bias accumulation. This embodiment designs a zero bias estimation algorithm based on static detection, and calculates the bias value by identifying the static state segment. The calibration process uses a multi-pose optimization method to establish a mapping relationship between the IMU output and the reference motion.

[0029] In the feature extraction stage, this embodiment extracts ORB features from the video sequence, including FAST corner detection and BRIEF descriptor calculation. Depth information is obtained through an improved binocular matching algorithm, which improves the accuracy of depth estimation. Point cloud feature extraction includes normal vector calculation based on principal curvature analysis and edge feature extraction based on curvature change.

[0030] The establishment of the time-space correspondence adopts a multi-level matching strategy. This embodiment first establishes the spatial correspondence of features and accelerates the nearest neighbor search through the KD tree. The time correspondence uses a sliding window method to consider sensor delay and sampling frequency differences. The feature association uses the RANSAC algorithm to eliminate false matches and improve the reliability of the correspondence.

[0031] Data fusion uses an improved particle filter algorithm. This embodiment designs a state transition model and observation model that adapt to the characteristics of multi-source data, and alleviates the particle degradation problem through an adaptive resampling strategy. The filtering process considers the uncertainty of various features and performs weighted fusion of information from different sources.

[0032] In practical applications, this embodiment achieves accurate geometric reconstruction and texture mapping in indoor scene reconstruction. In outdoor environments, it can adapt to lighting changes and occlusions and maintain the stability of feature extraction. For dynamic scenes, the tracking accuracy of moving targets is improved through multi-source data fusion.

[0033] This embodiment solves the problem of incomplete information in traditional single-source data collection methods. The comprehensiveness of scene perception is improved through multi-source data fusion, and the data quality of subsequent modeling is guaranteed through efficient preprocessing and feature extraction. This solution is particularly suitable for application scenarios that require accurate 3D reconstruction, such as indoor navigation, augmented reality, etc.

[0034] Step S102: constructing a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain the scene semantic type, performing octree partitioning on the point cloud data to calculate the spatial density coefficient, using a frustum projection method to statistically calculate the depth occlusion map of the multi-source scene information to obtain the scene complexity coefficient, constructing a geometric descriptor based on the normal vector feature and the edge feature to calculate the shape variation coefficient, performing nonlinear weighting on the density coefficient, the complexity coefficient and the shape variation coefficient to obtain a scene score, constructing a multimodal feature representation according to the scene semantic type and the scene score, and selecting a modeling algorithm and a parameter combination in a policy network trained by deep reinforcement learning; Optionally, this embodiment proposes a scene assessment method based on multimodal analysis, which realizes adaptive assessment of scene complexity and intelligent selection of modeling strategies by fusing semantic information and geometric features.

[0035] In the construction of the multimodal scene analysis network, this embodiment adopts an improved Transformer architecture and designs a multi-head attention mechanism to process feature information of different modalities. The network input includes image features, point cloud features, and motion feature sequences, and the spatial position information is retained through position encoding. The attention layer calculates the association weights between different features to achieve effective fusion of cross-modal features.

[0036] The scene semantic analysis uses the instance segmentation method. This embodiment connects the segmentation head network after the Transformer decoder to improve the segmentation accuracy through the multi-scale feature pyramid structure. The network output includes the category label and instance mask of the scene elements, which can distinguish different semantic types such as walls, floors, and furniture.

[0037] The spatial density evaluation adopts an adaptive octree partitioning strategy. This embodiment dynamically adjusts the partitioning granularity according to the local distribution characteristics of the point cloud, accurately expressing the scene details while maintaining computational efficiency. The density coefficient is normalized to reflect the geometric complexity of the scene.

[0038] The scene complexity evaluation adopts an improved cone projection method. This embodiment constructs a multi-view projection model to calculate the overlap and occlusion relationship of scene elements. The visual occlusion area is identified through depth map analysis to evaluate the spatial layout complexity of the scene.

[0039] Shape change analysis is based on local geometric features. This embodiment combines normal vectors and edge features to construct a multi-scale geometric descriptor, and describes local shape changes through main direction analysis and curvature statistics. The variance of the descriptor reflects the degree of geometric irregularity of the scene.

[0040] The scene scoring adopts a nonlinear weighted fusion method. This embodiment designs a multi-layer perceptron network to learn the nonlinear relationship between the three evaluation coefficients. The adaptive weights are calculated through the Softmax layer to achieve dynamic fusion of evaluation indicators. The scoring result comprehensively reflects the modeling difficulty of the scene.

[0041] The multimodal feature representation adopts a hybrid encoding method. In this embodiment, the scene semantic type is one-hot encoded and concatenated with the scene score feature to construct a joint representation. This representation retains the semantic information and complexity information of the scene, providing a basis for the subsequent modeling strategy selection.

[0042] The modeling strategy selection adopts the deep reinforcement learning method. This embodiment constructs an action space including algorithm selection and parameter adjustment, and designs a reward function based on reconstruction accuracy. The network parameters are optimized through the policy gradient algorithm to achieve online learning and optimization of the modeling strategy.

[0043] In practical applications, this embodiment can adaptively select a suitable modeling strategy according to the characteristics of the scene. For regular indoor scenes, a fast reconstruction algorithm based on plane features is preferred. For complex outdoor environments, a fine reconstruction method with stronger geometric constraints is selected.

[0044] This embodiment solves the problem that traditional fixed modeling strategies are difficult to adapt to complex scenes. Through multi-dimensional scene evaluation and intelligent strategy selection, the adaptability and robustness of 3D reconstruction are improved. This solution shows good modeling effects in applications such as large-scale scene reconstruction and augmented reality.

[0045] Step S103: using a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculating the topological structure change of the point cloud data to generate a deformation vector field, estimating the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence, inputting the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, applying the modeling algorithm and the parameter combination to the dynamic area for adaptive mesh subdivision and geometric constraint reconstruction, calculating a dynamic threshold according to the scene semantic type, and when the reconstruction error of the dynamic area exceeds the dynamic threshold, reselecting the modeling algorithm in the strategy network to reconstruct the dynamic area.

[0046] Optionally, this embodiment proposes a dynamic scene adaptive reconstruction method, which achieves accurate three-dimensional reconstruction of complex dynamic scenes through multi-source motion information analysis and intelligent algorithm selection.

[0047] In scene motion analysis, this embodiment designs a two-stream neural network structure to process RGB image sequences and depth map sequences respectively. The spatial stream network extracts the spatial features of the scene, and the temporal stream network captures the temporal change features. The information of the two branches is combined through the feature fusion layer to generate a three-dimensional vector field describing the scene motion.

[0048] The point cloud deformation analysis adopts the local structure tracking method. This embodiment calculates the neighborhood topology of each point by constructing a local connection relationship graph of the point cloud. Based on the structural changes between consecutive time frames, the local deformation of the point cloud is estimated. The deformation vector field reflects the non-rigid deformation characteristics of objects in the scene.

[0049] The camera motion estimation adopts a tightly coupled inertial vision fusion method. This embodiment combines the acceleration and angular velocity information of the IMU data with the visual feature matching results to construct a nonlinear optimization problem. Through the sliding window optimization method, real-time and accurate estimation of the camera pose is achieved.

[0050] Dynamic region prediction uses a graph neural network model. This embodiment encodes the 3D motion field, deformation vector field, and camera trajectory into node features of a graph structure, and extracts dynamic features through a graph convolution layer and an attention mechanism. The network outputs a probability map of the dynamic region to guide subsequent adaptive reconstruction.

[0051] Adaptive mesh subdivision is based on the octree structure. This embodiment adaptively adjusts the mesh granularity according to the spatial distribution and motion characteristics of the dynamic area. A more detailed mesh division is used in the area with intense motion to improve the reconstruction accuracy.

[0052] The geometric constraint reconstruction adopts a multi-view stereo vision method. This embodiment combines the modeling algorithm and parameters selected in the previous steps to construct geometric constraints that take into account temporal consistency. By minimizing the projection error and regularization term, the geometric accuracy of the reconstruction result is optimized.

[0053] Dynamic threshold calculation takes into account scene semantic information. This embodiment designs an adaptive threshold calculation strategy based on the geometric features and motion characteristics of different semantic types. For example, a looser threshold is used for non-rigid objects such as human bodies, while a stricter threshold is used for rigid objects such as buildings.

[0054] The reconstruction quality is evaluated by multi-scale error analysis. This embodiment evaluates the reconstruction accuracy by comparing the geometric deviation between the reconstruction result and the original point cloud. When the error exceeds the dynamic threshold, the strategy network is triggered to reselect a more suitable modeling algorithm.

[0055] In practical applications, this embodiment can effectively handle complex scenes containing multiple moving objects. For human motion in indoor scenes, the motion posture can be accurately reconstructed. For vehicle motion in outdoor scenes, the motion trajectory and shape changes can be accurately captured.

[0056] This embodiment solves the problem that traditional static reconstruction methods fail in dynamic scenes. Through multi-source motion analysis and adaptive reconstruction strategies, the accuracy and robustness of dynamic scene reconstruction are improved. This solution has important value in applications that require real-time scene understanding, such as robot navigation and augmented reality.

[0057] From the above description, it can be seen that the dynamic modeling method of the scene space three-dimensional model based on multimodal data provided in the embodiment of the present application can establish the time-space correspondence relationship by fusing the feature information of the video sequence, point cloud data and IMU data, and using the particle filter algorithm. The Transformer-based multimodal scene analysis network realizes scene semantic understanding, combines spatial density, depth occlusion and shape change features to calculate the scene score, and selects the optimal modeling algorithm through the policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0058] In one embodiment of the scene space three-dimensional model dynamic modeling method based on multimodal data of the present application, see Figure 2 , and can also include the following: Step S201: inputting the video sequence into a U-shaped convolutional neural network for multi-scale feature extraction, performing jump connection fusion on the feature map to obtain a noise distribution map, reconstructing the denoised video sequence based on the noise distribution map, calculating the positions of checkerboard corner points in the video sequence, constructing a nonlinear equation group based on the Zhang camera model, using the Levenberg-Marquardt algorithm to solve the equation group to obtain lens distortion parameters, and correcting the denoised video sequence according to the distortion parameters; Step S202: Perform FAST corner detection on the video sequence to obtain a feature point set, calculate the BRIEF descriptor of the feature point set to construct ORB features, use a binocular disparity algorithm to calculate a depth map, construct a K nearest neighbor graph for the point cloud data, calculate the covariance matrix of each point based on the K nearest neighbor graph to obtain a normal vector, use an edge detection operator to extract point cloud edge features, perform bandpass filtering on the IMU data to obtain an acceleration sequence, and perform gyroscope integration on the IMU data to obtain an angular velocity sequence.

[0059] Optionally, this embodiment proposes a multi-sensor data preprocessing and feature extraction method, which achieves high-quality visual, point cloud and inertial data processing by combining deep learning and geometric analysis.

[0060] In the video sequence denoising process, this embodiment adopts an improved U-Net network structure. The network consists of two main parts: an encoder and a decoder. The encoder extracts multi-scale features through multi-layer convolution and pooling operations, and the decoder gradually restores the image resolution through deconvolution operations. In the feature fusion process, a jump connection structure is used to directly transfer the low-level features of the encoder to the corresponding layer of the decoder to retain the image detail information. The network outputs a noise distribution map to guide the image denoising process.

[0061] The camera calibration adopts the improved Zhang calibration method. In this embodiment, the checkerboard corners are first detected in the video sequence, and the position accuracy is improved by the sub-pixel precision corner detection algorithm. The mapping relationship between the world coordinate system and the image coordinate system is established based on the homography matrix, and a nonlinear equation group containing camera intrinsic parameters and distortion parameters is constructed. The Levenberg-Marquardt optimization algorithm is used to solve the equation group to obtain the radial distortion and tangential distortion parameters of the lens.

[0062] The image feature extraction uses the ORB (Oriented FAST and Rotated BRIEF) algorithm. This embodiment first uses the FAST (Features from Accelerated Segment Test) algorithm to detect corner points, and selects the feature points with the strongest response value through non-maximum suppression. The main direction is calculated for each feature point to ensure that the feature is rotationally invariant. The BRIEF descriptor generates a binary string by comparing the grayscale values ​​of pixels in the neighborhood of the feature point, providing a compact and computationally efficient feature representation.

[0063] The depth information is acquired by using an improved binocular matching algorithm. This embodiment calculates the disparity map by using a local stereo matching method and optimizes the disparity estimation result by using dynamic programming. The image texture and edge information are considered to improve the accuracy of depth estimation in weak texture areas.

[0064] Point cloud feature extraction includes normal vector and edge feature calculation. This embodiment first constructs a local connection graph based on K nearest neighbors, and calculates the covariance matrix of each point through principal component analysis. The normal vector is determined by the eigenvector corresponding to the minimum eigenvalue of the covariance matrix. The edge feature is extracted by analyzing the local curvature change, which can effectively identify the geometric discontinuity area in the point cloud.

[0065] IMU data processing adopts a multi-level filtering strategy. This embodiment uses bandpass filtering on acceleration data to remove high-frequency noise and low-frequency drift. Angular velocity data is obtained through an improved numerical integration method, and zero bias estimation and temperature compensation are used to improve integration accuracy.

[0066] In indoor navigation scenarios, the data preprocessing method of this embodiment can effectively improve the stability of feature extraction. Through noise reduction and correction, the image quality in low-light environments is improved. The point cloud feature extraction method shows good robustness when processing reflective material surfaces.

[0067] This embodiment solves the problem of unstable quality of multi-source data. By combining deep learning and geometric analysis, the accuracy and robustness of feature extraction are improved. This solution provides a high-quality data foundation for subsequent scene reconstruction and motion estimation, and has wide application value in the fields of augmented reality, robot vision, etc.

[0068] In one embodiment of the scene space three-dimensional model dynamic modeling method based on multimodal data of the present application, see Figure 3 , and can also include the following: Step S301: construct a KD tree index structure for the ORB image features, calculate feature matching pairs based on nearest neighbor search, remove abnormal matching points using the RANSAC algorithm, project the depth information into three-dimensional space to establish a point cloud mapping relationship, perform spatial position indexing on the normal vector features and the edge features, establish a feature association table aligned with timestamps, and construct a feature spatiotemporal consistency verification model using camera pose and IMU motion constraints; Step S302: construct a state space equation based on the feature spatiotemporal consistency verification model, initialize the particle swarm state vector and weight, calculate the likelihood function between the observed data in the feature association table and the particle state, update the particle weight according to the likelihood function, perform a resampling process to eliminate weight degradation, and obtain the fused multi-source scene information based on the weighted average of the particle swarm state vector.

[0069] Optionally, this embodiment proposes a method for spatiotemporal correlation and fusion of multi-source scene information, which achieves efficient fusion of visual, point cloud and inertial data through feature index construction and particle filter optimization.

[0070] In the feature matching link, this embodiment first constructs a KD tree index structure for the ORB feature. By establishing a binary tree partition in the high-dimensional feature space, the nearest neighbor search of the feature point is accelerated. The feature matching adopts a ratio test strategy to screen reliable matching pairs by setting a distance threshold. The RANSAC algorithm removes abnormal matching points that do not meet the spatial transformation constraints through random sampling and consistency checks.

[0071] The point cloud mapping relationship is established by using a depth projection method. In this embodiment, the depth information of the image feature points is projected into a three-dimensional space through the camera intrinsic parameter matrix to establish a corresponding relationship between the two-dimensional image features and the three-dimensional point cloud. This mapping relationship provides spatial constraints for subsequent feature fusion.

[0072] The geometric feature index adopts a multi-level space partitioning strategy. This embodiment constructs an octree index structure for the normal vector features and edge features of the point cloud to support fast spatial neighborhood queries. By analyzing the spatial distribution characteristics of the features, the local connection relationship between the feature points is established.

[0073] The construction of the feature association table takes into account the time synchronization problem. This embodiment aligns different sensor data through hardware timestamps to establish a unified time reference. The association table records the timestamp, spatial position, descriptor and other information of the feature points, providing a spatiotemporal index for feature fusion.

[0074] The spatiotemporal consistency verification model combines multiple constraints. This embodiment uses the camera's motion model and the IMU's inertial constraints to build a physical model of feature motion. By analyzing the temporal changes and spatial distribution of feature points, the rationality of feature association is verified.

[0075] The state space equation describes the feature fusion process. In this embodiment, the state variables such as the position and velocity of the feature points are combined into a state vector to construct a nonlinear state equation that takes into account multi-sensor observations. The observation equation contains multiple constraints such as visual reprojection error and IMU pre-integration error.

[0076] The particle filter optimization adopts an adaptive resampling strategy. In this embodiment, the particle swarm is first initialized, and each particle represents a possible state hypothesis. The weight of each particle is evaluated by calculating the likelihood function between the observed data and the particle state. When the weight distribution is degraded, the resampling process is performed to maintain the particle diversity.

[0077] In indoor scene reconstruction applications, this embodiment can effectively handle the problems of sensor data asynchrony and measurement noise. For fast motion scenes, the stability of feature tracking is improved through IMU constraints. In environments with changing lighting, geometric features provide additional matching constraints to enhance the robustness of feature association.

[0078] This embodiment solves the problem of inaccurate feature association and unstable state estimation in multi-source data fusion. Through efficient feature index structure and particle filter optimization, the accuracy and real-time performance of feature fusion are improved. This solution shows significant advantages in applications that require accurate scene understanding, such as visual SLAM and augmented reality.

[0079] In one embodiment of the scene space three-dimensional model dynamic modeling method based on multimodal data of the present application, see Figure 4, and can also include the following: Step S401: Mapping the multi-source scene information into a feature sequence, position encoding the feature sequence, calculating the association weights between features through a multi-head self-attention mechanism, transforming the weighted features using a feedforward neural network, constructing a semantic segmentation head network to classify scene elements, and generating a scene semantic type label map; Step S402: recursively divide the point cloud space into octrees, count the number of point clouds in each leaf node to calculate the density coefficient, build a camera frustum model to project the scene, calculate the projection overlap area to obtain the complexity coefficient, combine the normal vector feature and the edge feature to construct a local geometric descriptor, and calculate the variance of the descriptor to obtain the shape change coefficient.

[0080] Optionally, this embodiment proposes a scene semantic understanding and complexity assessment method based on deep learning, which achieves accurate semantic segmentation and feature description of the scene through feature sequence analysis and multi-dimensional feature fusion.

[0081] In the feature sequence processing stage, this embodiment first converts the multi-source scene information into a unified feature representation. The spatial position information of the feature is encoded into the feature vector using the sinusoidal position encoding method. The position encoding contains sinusoidal functions of different frequencies, which enables the network to perceive the relative position relationship of the features.

[0082] The multi-head self-attention mechanism realizes the global association of features. In this embodiment, the feature sequence is input into multiple parallel attention heads. Each attention head independently calculates the query and key-value matrix, and calculates the association weights between features by scaling the dot product attention. The multi-head mechanism allows the network to learn feature relationships from different representation subspaces, enhancing the expressiveness of feature extraction.

[0083] The feedforward neural network performs feature transformation. This embodiment adopts a two-layer fully connected network structure and enhances the network's expressiveness through a nonlinear activation function. Residual connections and layer normalization are used to stabilize the training process and improve network convergence performance.

[0084] The semantic segmentation head network adopts a multi-scale feature fusion strategy. This embodiment designs a feature pyramid structure to fuse feature information at different levels through upsampling and skip connections. The decoder gradually restores the feature resolution and finally outputs a semantic label map of the same size as the input image.

[0085] The scene complexity evaluation adopts multi-dimensional feature analysis. In this embodiment, the point cloud space is first adaptively divided by recursive octree. The number of point clouds is counted in each leaf node, and the local point cloud density coefficient is calculated to reflect the geometric complexity of the scene.

[0086] The frustum projection analysis considers the visibility of the scene. This embodiment constructs a camera frustum model and calculates the projection overlap area of ​​the scene under different viewing angles. The area and depth changes of the overlapping area are used to evaluate the observation complexity of the scene.

[0087] The local geometric description adopts a multi-feature fusion method. In this embodiment, the normal vector feature and the edge feature are combined to construct a descriptor, which contains geometric information such as the local curvature and the direction of the main curvature of the point cloud. The degree of shape change of the scene is quantified by calculating the variance of the descriptor in the local neighborhood.

[0088] In indoor scene understanding applications, this embodiment can accurately identify different semantic elements such as furniture and walls. For areas with complex geometric structures, such as stairwells and multi-layer bookshelves, the scene complexity can be accurately assessed to provide a reference for subsequent reconstruction.

[0089] This embodiment solves the problem that traditional scene understanding methods have poor adaptability to complex scenes. By combining deep learning and geometric analysis, the accuracy and robustness of scene semantic understanding are improved. This solution has important value in applications such as robot environment perception and intelligent scene reconstruction, and can provide reliable semantic information and complexity assessment for subsequent scene modeling.

[0090] In one embodiment of the scene space three-dimensional model dynamic modeling method based on multimodal data of the present application, see Figure 5 , and can also include the following: Step S501: perform feature mapping on the density coefficient, the complexity coefficient and the shape change coefficient using a multi-layer perceptron network, input the mapping result into a Softmax layer to calculate a weight coefficient, perform weighted summation on the three coefficients based on the weight coefficient to obtain a scene score, perform one-hot encoding on the scene semantic type, and concatenate the scene score with the encoding result to construct a multimodal feature vector; Step S502: Use the multimodal feature vector as the state input of the policy network, construct an action space including modeling algorithm selection and parameter adjustment, use the policy gradient algorithm to optimize the network parameters, design a reward function based on reconstruction accuracy, select the modeling algorithm by exploring and utilizing a balanced strategy, and determine the corresponding parameter combination according to the reward function.

[0091] Optionally, this embodiment proposes a scene adaptive modeling method based on deep reinforcement learning, which realizes the intelligent selection of modeling algorithms and parameters through multimodal feature analysis and strategy optimization.

[0092] The feature mapping adopts a multi-layer perceptron network structure. This embodiment designs a three-layer fully connected network, and uses the ReLU activation function between each layer to increase the nonlinear expression ability. The network performs feature transformation on the density coefficient, complexity coefficient and shape change coefficient respectively, and maps features of different scales to a unified feature space.

[0093] The weight calculation uses Softmax normalization. This embodiment converts the feature mapping result into a probability distribution through the Softmax layer to ensure that the sum of the weights of the three coefficients is 1. The weight coefficient reflects the importance of different features in the scene scoring and can adaptively adjust the contribution of each feature.

[0094] Scene scoring integrates multi-dimensional feature information. This embodiment comprehensively considers the geometric density, observation complexity and shape change characteristics of the scene by weighted summation. The scoring result quantifies the overall modeling difficulty of the scene and provides a basis for subsequent algorithm selection.

[0095] Semantic information encoding adopts one-hot encoding. In this embodiment, the scene semantic type is converted into a binary vector, and each position corresponds to a semantic category. By splicing the scene score and semantic encoding, a multimodal feature vector containing geometric and semantic information is constructed.

[0096] The policy network uses an Actor-Critic architecture. This embodiment uses a multimodal feature vector as the state input, and the network outputs the probability distribution selected by the modeling algorithm and the continuous value of the parameter adjustment. The Actor network is responsible for generating actions, and the Critic network evaluates the state value function.

[0097] Action space design considers algorithm selection and parameter adjustment. In this embodiment, commonly used 3D reconstruction algorithms (such as TSDF fusion, Poisson reconstruction, etc.) are selected as discrete actions, and key parameters (such as voxel size, smoothing factor, etc.) are mapped into a continuous action space.

[0098] The PPO (Proximal Policy Optimization) algorithm is used for policy optimization. This embodiment stabilizes the policy update process through importance sampling and trust region constraints. The generalized advantage function is used for gradient estimation to reduce the variance of policy updates.

[0099] The reward function design is based on reconstruction quality evaluation. This embodiment comprehensively considers reconstruction accuracy, completeness and geometric consistency to construct a multi-objective reward function. Different weight coefficients are set for scene areas of different semantic types to guide the network to learn scene adaptive modeling strategies.

[0100] In practical applications, this embodiment can automatically select a suitable modeling method according to scene features. For flat wall areas, a parametric reconstruction algorithm is preferred; for complex object surfaces, a voxel-based fine reconstruction method is selected.

[0101] This embodiment solves the problem that traditional fixed parameter modeling methods are difficult to adapt to complex scenes. Through the deep reinforcement learning method, the online optimization of the modeling strategy is realized, and the adaptability and robustness of 3D reconstruction are improved. This solution has important value in applications such as large-scale scene reconstruction and intelligent modeling, and can significantly improve the efficiency and quality of reconstruction.

[0102] In one embodiment of the scene space three-dimensional model dynamic modeling method based on multimodal data of the present application, see Figure 6 , and can also include the following: Step S601: input the video sequence and the depth information into a three-dimensional convolutional neural network to extract spatiotemporal features, restore the scene spatial resolution through a deconvolution layer, generate a dense optical flow field, construct a topological adjacency graph for the point cloud data, calculate the displacement difference of the point cloud nodes at adjacent times, and generate a vector field describing the local deformation; Step S602: Perform gravity compensation and integral calculation of displacement on the acceleration sequence, integrate the angular velocity sequence to obtain the rotation quaternion, construct an extended Kalman filter state equation, use the displacement and the rotation quaternion as observation quantities, recursively estimate the six-degree-of-freedom pose parameters of the camera based on the kinematic model, and generate a camera motion trajectory.

[0103] Optionally, this embodiment proposes a scene dynamic feature extraction and camera trajectory estimation method based on multi-source data fusion, which realizes accurate reconstruction and positioning of dynamic scenes through the combination of deep learning and filtering optimization.

[0104] The spatiotemporal feature extraction uses a three-dimensional convolution structure. This embodiment designs a multi-scale three-dimensional convolutional neural network, which simultaneously processes the spatial and temporal dimension information of the video sequence through a 3D convolution kernel. The network contains multiple residual modules, each of which uses spatial and temporal separation of convolution operations to improve the computational efficiency of feature extraction.

[0105] The dense optical flow field generation adopts a multi-layer deconvolution architecture. This embodiment integrates feature information at different levels through jump connections to gradually restore the spatial resolution of the scene. In the deconvolution process, the dilated convolution is introduced to expand the receptive field and enhance the network's modeling ability for large displacement motion.

[0106] Point cloud deformation analysis is based on graph structure representation. This embodiment first constructs a K-nearest neighbor topology graph of the point cloud, where each node contains the three-dimensional coordinates of the point and a local feature description. By calculating the displacement difference of the corresponding nodes at adjacent moments, a vector field describing the local deformation of the scene is generated.

[0107] IMU data preprocessing eliminates the influence of gravity. This embodiment obtains the direction of gravity through attitude estimation and performs gravity compensation on the acceleration data. After zero bias correction and integration processing, displacement information describing the camera movement is obtained.

[0108] The rotation parameter calculation is represented by quaternion. This embodiment numerically integrates the angular velocity sequence and updates the rotation state of the camera by quaternion multiplication. The quaternion representation avoids the gimbal lock problem of Euler angles and ensures the continuity of attitude estimation.

[0109] The extended Kalman filter design considers multi-source observations. This embodiment uses the camera position, velocity, attitude and IMU bias as state vectors to construct a kinematic model considering IMU pre-integration. The observation equation combines visual reprojection error and IMU measurement constraints to achieve robust state estimation.

[0110] The recursive estimation process uses an adaptive update strategy. This embodiment dynamically adjusts the Kalman gain according to the statistical characteristics of the observation noise to balance the credibility of the prediction model and the observation data. When large movements or rapid turns occur, the weight of the IMU observation is increased to improve tracking stability.

[0111] In dynamic scene reconstruction applications, this embodiment can accurately capture moving objects in the scene. For dynamic elements such as human motion and object deformation, accurate motion description is provided through optical flow field and deformation vector field. At the same time, stable camera trajectory estimation provides a reliable perspective reference for dynamic reconstruction.

[0112] This embodiment solves the adaptability problem of traditional static reconstruction methods in dynamic scenes. Through multi-source data fusion and deep learning methods, the accuracy and robustness of dynamic scene understanding are improved. This solution is of great value in applications that require real-time dynamic reconstruction, such as augmented reality and robot navigation, and can provide accurate dynamic features and camera pose information for subsequent scene reconstruction and interaction.

[0113] In one embodiment of the scene space three-dimensional model dynamic modeling method based on multimodal data of the present application, see Figure 7 , and can also include the following: Step S701: construct a graph convolutional network to perform feature aggregation on the three input fields, capture the temporal variation features through the gated recurrent unit, use the graph pooling layer to output the dynamic region probability map, determine the dynamic region boundary based on the probability map, perform quadtree meshing within the dynamic region, and use the modeling algorithm to geometrically reconstruct the mesh nodes; Step S702: setting a benchmark threshold according to the scene semantic type, dynamically adjusting the benchmark threshold based on the scene complexity, calculating the geometric error between the reconstructed point cloud and the original point cloud of the dynamic area, comparing the geometric error with the dynamic threshold, and triggering the policy network to re-execute the modeling algorithm selection.

[0114] Optionally, this embodiment first processes the input optical flow field, deformation vector field and probability field through a graph convolutional network. During the feature aggregation process, each node exchanges information with its neighboring nodes through a message passing mechanism to achieve a gradual fusion of features from local to global. After multiple layers of graph convolution operations, the network can effectively capture the spatial correlation in the scene.

[0115] Based on feature extraction, this embodiment introduces a recurrent neural network structure to analyze the dynamic changes of the scene. Through forward and backward bidirectional processing, the network can make full use of the information of historical and future frames to accurately capture the temporal dependencies in the scene. The selective update mechanism ensures that important dynamic information is retained.

[0116] This embodiment generates a dynamic region probability map through a hierarchical feature clustering method. The network gradually reduces the spatial resolution of the feature map during the pooling process while retaining significant dynamic features. Based on the generated probability map, a region growing algorithm is used to determine the boundary contour of the dynamic region.

[0117] This embodiment performs adaptive meshing in a determined dynamic region. The meshing process uses a quadtree structure to recursively subdivide the mesh according to the geometric features of the local region. In regions with complex boundaries or rich details, the meshing is more finely divided to ensure the accuracy of reconstruction.

[0118] This embodiment applies the selected modeling algorithm to the mesh nodes for geometric reconstruction. During the reconstruction process, the depth information and point cloud data are integrated, and corresponding reconstruction strategies are adopted for different types of surfaces. For example, plane fitting is adopted for flat areas, and implicit function reconstruction is used for complex curved surfaces.

[0119] This embodiment sets a baseline threshold for reconstruction quality assessment based on the semantic type of the scene. Different semantic types have different geometric features and reconstruction requirements, and therefore require differentiated quality assessment standards. For example, the wall area requires a higher level of flatness, while the object surface pays more attention to detail preservation.

[0120] This embodiment introduces a scene complexity adaptive mechanism to dynamically adjust the threshold. Complexity evaluation takes into account factors such as local curvature changes and depth discontinuity. In areas with high complexity, over-refinement and reconstruction are avoided by increasing the threshold tolerance.

[0121] This embodiment performs a multi-dimensional quality assessment on the reconstruction results. The assessment indicators include the geometric distance between point clouds, the consistency of normal vectors, etc. When the assessment result exceeds the dynamic threshold, the strategy network is triggered to reselect the algorithm and optimize the parameters.

[0122] In indoor scene reconstruction applications, this embodiment can accurately identify and reconstruct dynamic objects. For example, for dynamic elements such as moving human bodies, opening and closing doors and windows, the system can capture their motion features and accurately reconstruct them. At the same time, for static building structures, the reconstruction quality is maintained stable.

[0123] This embodiment solves the problem that traditional reconstruction methods are difficult to adapt to dynamic scenes. Through dynamic feature analysis and adaptive quality control, the accuracy and robustness of scene reconstruction are significantly improved. This technical solution has important value in real-time 3D reconstruction, augmented reality and other application fields, providing users with high-quality dynamic scene reconstruction services.

[0124] In order to effectively improve the quality and efficiency of 3D reconstruction of large-scale complex scenes, the present application provides an embodiment of a scene space 3D model dynamic modeling device based on multimodal data for realizing all or part of the content of the scene space 3D model dynamic modeling method based on multimodal data, see Figure 8 The scene space three-dimensional model dynamic modeling device based on multimodal data specifically includes the following contents: The multi-source data processing module 10 is used to collect video sequences, point cloud data and IMU data of large-scale indoor and outdoor scenes, perform deep learning noise reduction and distortion correction on the video sequence, perform spatial downsampling and denoising on the point cloud data, perform zero bias calibration and calibration on the IMU data, extract ORB image features and depth information from the video sequence, extract normal vector features and edge features from the point cloud data, extract acceleration sequence and angular velocity sequence from the IMU data, establish a spatiotemporal correspondence between the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence and the angular velocity sequence, and use a particle filter algorithm to fuse the correspondence to obtain multi-source scene information; A scene space processing module 20 is used to construct a Transformer-based multimodal scene analysis network, perform instance segmentation on the multi-source scene information to obtain a scene semantic type, perform octree partitioning on the point cloud data to calculate a spatial density coefficient, use a frustum projection method to statistically calculate a depth occlusion map of the multi-source scene information to obtain a scene complexity coefficient, construct a geometric descriptor based on the normal vector feature and the edge feature to calculate a shape variation coefficient, perform nonlinear weighting on the density coefficient, the complexity coefficient and the shape variation coefficient to obtain a scene score, construct a multimodal feature representation according to the scene semantic type and the scene score, and select a modeling algorithm and a parameter combination in a policy network trained by deep reinforcement learning; The dynamic modeling module 30 is used to use a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculate the topological structure change of the point cloud data to generate a deformation vector field, estimate the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence, input the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, apply the modeling algorithm and the parameter combination to the dynamic area for adaptive mesh subdivision and geometric constraint reconstruction, calculate a dynamic threshold according to the scene semantic type, and when the reconstruction error of the dynamic area exceeds the dynamic threshold, reselect the modeling algorithm in the strategy network to reconstruct the dynamic area.

[0125] From the above description, it can be seen that the scene space three-dimensional model dynamic modeling device based on multimodal data provided in the embodiment of the present application can establish a spatiotemporal correspondence relationship by fusing the feature information of video sequences, point cloud data and IMU data, and using a particle filter algorithm. The Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates scene scores based on spatial density, depth occlusion and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0126] From the hardware level, in order to effectively improve the quality and efficiency of 3D reconstruction of large-scale complex scenes, the present application provides an embodiment of an electronic device for implementing all or part of the content of the method for dynamic modeling of a scene space 3D model based on multimodal data, and the electronic device specifically includes the following content: Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the dynamic modeling device of scene space three-dimensional model based on multimodal data and related devices such as core business system, user terminal and related database; the logic controller can be a desktop computer, tablet computer and mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the dynamic modeling method of scene space three-dimensional model based on multimodal data and the embodiment of the dynamic modeling device of scene space three-dimensional model based on multimodal data in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0127] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0128] In practical applications, part of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data can be executed on the electronic device side as described above, or all operations can be completed in the client device. The selection can be made based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0129] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0130] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0131] In one embodiment, the function of the scene space three-dimensional model dynamic modeling method based on multimodal data can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control: Step S101: collecting video sequences, point cloud data and IMU data of large-scale indoor and outdoor scenes, performing deep learning noise reduction and distortion correction on the video sequences, performing spatial downsampling and denoising on the point cloud data, performing zero bias calibration and calibration on the IMU data, extracting ORB image features and depth information from the video sequences, extracting normal vector features and edge features from the point cloud data, extracting acceleration sequences and angular velocity sequences from the IMU data, establishing a spatiotemporal correspondence between the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence and the angular velocity sequence, and using a particle filter algorithm to fuse the correspondence to obtain multi-source scene information; Step S102: constructing a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain the scene semantic type, performing octree partitioning on the point cloud data to calculate the spatial density coefficient, using a frustum projection method to statistically calculate the depth occlusion map of the multi-source scene information to obtain the scene complexity coefficient, constructing a geometric descriptor based on the normal vector feature and the edge feature to calculate the shape variation coefficient, performing nonlinear weighting on the density coefficient, the complexity coefficient and the shape variation coefficient to obtain a scene score, constructing a multimodal feature representation according to the scene semantic type and the scene score, and selecting a modeling algorithm and a parameter combination in a policy network trained by deep reinforcement learning; Step S103: using a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculating the topological structure change of the point cloud data to generate a deformation vector field, estimating the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence, inputting the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, applying the modeling algorithm and the parameter combination to the dynamic area for adaptive mesh subdivision and geometric constraint reconstruction, calculating a dynamic threshold according to the scene semantic type, and when the reconstruction error of the dynamic area exceeds the dynamic threshold, reselecting the modeling algorithm in the strategy network to reconstruct the dynamic area.

[0132] From the above description, it can be seen that the electronic device provided in the embodiment of the present application uses a particle filter algorithm to establish a time-space correspondence relationship by fusing the feature information of video sequences, point cloud data and IMU data. The Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates scene scores based on spatial density, depth occlusion and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0133] In another embodiment, the scene space three-dimensional model dynamic modeling device based on multimodal data can be configured separately from the central processing unit 9100. For example, the scene space three-dimensional model dynamic modeling device based on multimodal data can be configured as a chip connected to the central processing unit 9100, and the function of the scene space three-dimensional model dynamic modeling method based on multimodal data can be implemented through the control of the central processing unit.

[0134] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0135] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0136] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0137] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0138] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0139] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0140] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0141] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0142] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps in the method for dynamically modeling a scene space three-dimensional model based on multimodal data, in which the execution subject is a server or a client in the above embodiments. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps in the method for dynamically modeling a scene space three-dimensional model based on multimodal data, in which the execution subject is a server or a client in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented: Step S101: collecting video sequences, point cloud data and IMU data of large-scale indoor and outdoor scenes, performing deep learning noise reduction and distortion correction on the video sequences, performing spatial downsampling and denoising on the point cloud data, performing zero bias calibration and calibration on the IMU data, extracting ORB image features and depth information from the video sequences, extracting normal vector features and edge features from the point cloud data, extracting acceleration sequences and angular velocity sequences from the IMU data, establishing a spatiotemporal correspondence between the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence and the angular velocity sequence, and using a particle filter algorithm to fuse the correspondence to obtain multi-source scene information; Step S102: constructing a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain the scene semantic type, performing octree partitioning on the point cloud data to calculate the spatial density coefficient, using a frustum projection method to statistically calculate the depth occlusion map of the multi-source scene information to obtain the scene complexity coefficient, constructing a geometric descriptor based on the normal vector feature and the edge feature to calculate the shape variation coefficient, performing nonlinear weighting on the density coefficient, the complexity coefficient and the shape variation coefficient to obtain a scene score, constructing a multimodal feature representation according to the scene semantic type and the scene score, and selecting a modeling algorithm and a parameter combination in a policy network trained by deep reinforcement learning; Step S103: using a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculating the topological structure change of the point cloud data to generate a deformation vector field, estimating the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence, inputting the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, applying the modeling algorithm and the parameter combination to the dynamic area for adaptive mesh subdivision and geometric constraint reconstruction, calculating a dynamic threshold according to the scene semantic type, and when the reconstruction error of the dynamic area exceeds the dynamic threshold, reselecting the modeling algorithm in the strategy network to reconstruct the dynamic area.

[0143] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application uses a particle filter algorithm to establish a spatiotemporal correspondence by fusing the feature information of video sequences, point cloud data, and IMU data. The Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates scene scores based on spatial density, depth occlusion, and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0144] The embodiments of the present application also provide a computer program product capable of implementing all the steps in the method for dynamically modeling a scene space three-dimensional model based on multimodal data in the above-mentioned embodiment, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the method for dynamically modeling a scene space three-dimensional model based on multimodal data are implemented. For example, the computer program / instruction implements the following steps: Step S101: collecting video sequences, point cloud data and IMU data of large-scale indoor and outdoor scenes, performing deep learning noise reduction and distortion correction on the video sequences, performing spatial downsampling and denoising on the point cloud data, performing zero bias calibration and calibration on the IMU data, extracting ORB image features and depth information from the video sequences, extracting normal vector features and edge features from the point cloud data, extracting acceleration sequences and angular velocity sequences from the IMU data, establishing a spatiotemporal correspondence between the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence and the angular velocity sequence, and using a particle filter algorithm to fuse the correspondence to obtain multi-source scene information; Step S102: constructing a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain the scene semantic type, performing octree partitioning on the point cloud data to calculate the spatial density coefficient, using a frustum projection method to statistically calculate the depth occlusion map of the multi-source scene information to obtain the scene complexity coefficient, constructing a geometric descriptor based on the normal vector feature and the edge feature to calculate the shape variation coefficient, performing nonlinear weighting on the density coefficient, the complexity coefficient and the shape variation coefficient to obtain a scene score, constructing a multimodal feature representation according to the scene semantic type and the scene score, and selecting a modeling algorithm and a parameter combination in a policy network trained by deep reinforcement learning; Step S103: using a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculating the topological structure change of the point cloud data to generate a deformation vector field, estimating the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence, inputting the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, applying the modeling algorithm and the parameter combination to the dynamic area for adaptive mesh subdivision and geometric constraint reconstruction, calculating a dynamic threshold according to the scene semantic type, and when the reconstruction error of the dynamic area exceeds the dynamic threshold, reselecting the modeling algorithm in the strategy network to reconstruct the dynamic area.

[0145] From the above description, it can be seen that the computer program product provided in the embodiment of the present application establishes a spatiotemporal correspondence by fusing the feature information of video sequences, point cloud data, and IMU data, and adopts a particle filter algorithm. The Transformer-based multimodal scene analysis network realizes scene semantic understanding, calculates scene scores by combining spatial density, depth occlusion, and shape change features, and selects the optimal modeling algorithm through a policy network trained by deep reinforcement learning. At the same time, deep neural networks and graph neural networks are used to analyze the dynamic characteristics of the scene, realize adaptive mesh subdivision and geometric constraint reconstruction of dynamic areas, and dynamically adjust the reconstruction threshold according to the scene semantic type. This method effectively improves the quality and efficiency of three-dimensional reconstruction of large-scale complex scenes.

[0146] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0150] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for dynamic modeling of a scene space three-dimensional model based on multimodal data, characterized in that: The method comprises: Collect video sequences, point cloud data and IMU data of large-scale indoor and outdoor scenes, perform deep learning noise reduction and distortion correction on the video sequences, perform spatial downsampling and denoising on the point cloud data, perform zero bias calibration and calibration on the IMU data, extract ORB image features and depth information from the video sequences, extract normal vector features and edge features from the point cloud data, extract acceleration sequences and angular velocity sequences from the IMU data, establish a spatiotemporal correspondence between the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence and the angular velocity sequence, and use a particle filter algorithm to fuse the correspondence to obtain multi-source scene information; Construct a Transformer-based multimodal scene analysis network, perform instance segmentation on the multi-source scene information to obtain the scene semantic type, perform octree partitioning on the point cloud data to calculate the spatial density coefficient, use the cone projection method to statistically calculate the depth occlusion map of the multi-source scene information to obtain the scene complexity coefficient, construct a geometric descriptor based on the normal vector feature and the edge feature to calculate the shape change coefficient, perform nonlinear weighting on the density coefficient, the complexity coefficient and the shape change coefficient to obtain a scene score, construct a multimodal feature representation according to the scene semantic type and the scene score, and select a modeling algorithm and parameter combination in the policy network trained by deep reinforcement learning; The video sequence and the depth information are fused by using a deep neural network to obtain a three-dimensional motion field of the scene, the topological structure change of the point cloud data is calculated to generate a deformation vector field, the camera motion trajectory is estimated based on the fusion of the acceleration sequence and the angular velocity sequence, the three-dimensional motion field, the deformation vector field and the motion trajectory are input into a graph neural network to predict the dynamic area of ​​the scene, the modeling algorithm and the parameter combination are applied to the dynamic area to perform adaptive mesh subdivision and geometric constraint reconstruction, the dynamic threshold is calculated according to the scene semantic type, and when the reconstruction error of the dynamic area exceeds the dynamic threshold, the modeling algorithm is reselected in the strategy network to reconstruct the dynamic area.

2. The method for dynamic modeling of scene space three-dimensional model based on multimodal data according to claim 1, characterized in that: The method includes performing deep learning noise reduction and distortion correction on the video sequence, performing spatial downsampling and denoising on the point cloud data, performing zero bias calibration and calibration on the IMU data, extracting ORB image features and depth information from the video sequence, extracting normal vector features and edge features from the point cloud data, and extracting acceleration sequences and angular velocity sequences from the IMU data, including: The video sequence is input into a U-shaped convolutional neural network for multi-scale feature extraction, the feature map is subjected to skip connection fusion to obtain a noise distribution map, the denoised video sequence is reconstructed based on the noise distribution map, the checkerboard corner point positions in the video sequence are calculated, a nonlinear equation group based on the Zhang camera model is constructed, the Levenberg-Marquardt algorithm is used to solve the equation group to obtain lens distortion parameters, and the denoised video sequence is corrected according to the distortion parameters; Perform FAST corner detection on the video sequence to obtain a feature point set, calculate the BRIEF descriptor of the feature point set to construct ORB features, use a binocular disparity algorithm to calculate a depth map, construct a K nearest neighbor graph for the point cloud data, calculate the covariance matrix of each point based on the K nearest neighbor graph to obtain a normal vector, use an edge detection operator to extract point cloud edge features, perform bandpass filtering on the IMU data to obtain an acceleration sequence, and perform gyroscope integration on the IMU data to obtain an angular velocity sequence.

3. The method for dynamic modeling of scene space three-dimensional model based on multimodal data according to claim 1, characterized in that: The step of establishing a spatiotemporal correspondence between the ORB image feature, the depth information, the normal vector feature, the edge feature, the acceleration sequence, and the angular velocity sequence, and fusing the correspondence using a particle filter algorithm to obtain multi-source scene information includes: Construct a KD tree index structure for the ORB image features, calculate feature matching pairs based on nearest neighbor search, remove abnormal matching points using the RANSAC algorithm, project the depth information into three-dimensional space to establish a point cloud mapping relationship, perform spatial position indexing on the normal vector features and the edge features, establish a feature association table aligned with timestamps, and construct a feature spatiotemporal consistency verification model using camera pose and IMU motion constraints; A state space equation based on the feature spatiotemporal consistency verification model is constructed, the particle swarm state vector and weight are initialized, the likelihood function between the observed data in the feature association table and the particle state is calculated, the particle weight is updated according to the likelihood function, a resampling process is performed to eliminate weight degradation, and the fused multi-source scene information is obtained based on the weighted average value of the particle swarm state vector.

4. The method for dynamic modeling of scene space three-dimensional model based on multimodal data according to claim 1, characterized in that: The method comprises: constructing a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain a scene semantic type, performing octree partitioning on the point cloud data to calculate a spatial density coefficient, using a frustum projection method to calculate a depth occlusion map of the multi-source scene information to obtain a scene complexity coefficient, and constructing a geometric descriptor based on the normal vector feature and the edge feature to calculate a shape change coefficient, including: Mapping the multi-source scene information into a feature sequence, encoding the feature sequence by position, calculating the association weights between features through a multi-head self-attention mechanism, transforming the weighted features using a feedforward neural network, constructing a semantic segmentation head network to classify scene elements, and generating a scene semantic type label map; The point cloud space is recursively divided into octrees, the number of point clouds in each leaf node is counted to calculate the density coefficient, a camera frustum model is constructed to project the scene, the projection overlapping area is calculated to obtain the complexity coefficient, the normal vector feature and the edge feature are combined to construct a local geometric descriptor, and the variance of the descriptor is calculated to obtain the shape change coefficient.

5. The method for dynamic modeling of scene space three-dimensional model based on multimodal data according to claim 1, characterized in that: The step of nonlinearly weighting the density coefficient, the complexity coefficient, and the shape change coefficient to obtain a scene score, constructing a multimodal feature representation according to the scene semantic type and the scene score, and selecting a modeling algorithm and a parameter combination in a policy network trained by deep reinforcement learning includes: Using a multi-layer perceptron network to perform feature mapping on the density coefficient, the complexity coefficient and the shape change coefficient, inputting the mapping result into a Softmax layer to calculate a weight coefficient, performing weighted summation on the three coefficients based on the weight coefficient to obtain a scene score, performing one-hot encoding on the scene semantic type, and concatenating the scene score with the encoding result to construct a multimodal feature vector; The multimodal feature vector is used as the state input of the policy network, an action space including modeling algorithm selection and parameter adjustment is constructed, the policy gradient algorithm is used to optimize the network parameters, a reward function is designed based on the reconstruction accuracy, the modeling algorithm is selected by exploring and utilizing the balance strategy, and the corresponding parameter combination is determined according to the reward function.

6. The method for dynamic modeling of scene space three-dimensional model based on multimodal data according to claim 1, characterized in that: The method of using a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculating the topological structure change of the point cloud data to generate a deformation vector field, and estimating the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence includes: Input the video sequence and the depth information into a three-dimensional convolutional neural network to extract spatiotemporal features, restore the scene spatial resolution through a deconvolution layer, generate a dense optical flow field, construct a topological adjacency graph for the point cloud data, calculate the displacement difference of the point cloud nodes at adjacent times, and generate a vector field describing the local deformation; The acceleration sequence is gravity compensated and integrated to calculate the displacement, the angular velocity sequence is integrated to obtain the rotation quaternion, an extended Kalman filter state equation is constructed, the displacement and the rotation quaternion are used as observation quantities, the six-degree-of-freedom posture parameters of the camera are recursively estimated based on the kinematic model, and the camera motion trajectory is generated.

7. The method for dynamic modeling of scene space three-dimensional model based on multimodal data according to claim 1, characterized in that: The method of inputting the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict a scene dynamic region, applying the modeling algorithm and the parameter combination to the dynamic region for adaptive mesh subdivision and geometric constraint reconstruction, calculating a dynamic threshold according to the scene semantic type, and reselecting a modeling algorithm in the strategy network to reconstruct the dynamic region when a reconstruction error of the dynamic region exceeds the dynamic threshold, comprises: Construct a graph convolutional network to aggregate the features of the three input fields, capture the temporal variation features through the gated recurrent unit, use the graph pooling layer to output the dynamic region probability map, determine the dynamic region boundary based on the probability map, perform quadtree meshing within the dynamic region, and use the modeling algorithm to geometrically reconstruct the mesh nodes; A benchmark threshold is set according to the scene semantic type, the benchmark threshold is dynamically adjusted based on the scene complexity, a geometric error between the reconstructed point cloud and the original point cloud of the dynamic area is calculated, the geometric error is compared with the dynamic threshold, and the policy network is triggered to re-execute the modeling algorithm selection.

8. A scene space three-dimensional model dynamic modeling device based on multimodal data, characterized in that: The device comprises: A multi-source data processing module is used to collect video sequences, point cloud data and IMU data of large-scale indoor and outdoor scenes, perform deep learning noise reduction and distortion correction on the video sequence, perform spatial downsampling and denoising on the point cloud data, perform zero bias calibration and calibration on the IMU data, extract ORB image features and depth information from the video sequence, extract normal vector features and edge features from the point cloud data, extract acceleration sequence and angular velocity sequence from the IMU data, establish a spatiotemporal correspondence between the ORB image features, the depth information, the normal vector features, the edge features, the acceleration sequence and the angular velocity sequence, and use a particle filter algorithm to fuse the correspondence to obtain multi-source scene information; A scene space processing module, for constructing a Transformer-based multimodal scene analysis network, performing instance segmentation on the multi-source scene information to obtain a scene semantic type, performing octree partitioning on the point cloud data to calculate a spatial density coefficient, using a frustum projection method to statistically calculate a depth occlusion map of the multi-source scene information to obtain a scene complexity coefficient, constructing a geometric descriptor based on the normal vector feature and the edge feature to calculate a shape variation coefficient, performing nonlinear weighting on the density coefficient, the complexity coefficient and the shape variation coefficient to obtain a scene score, constructing a multimodal feature representation according to the scene semantic type and the scene score, and selecting a modeling algorithm and a parameter combination in a policy network trained by deep reinforcement learning; A dynamic modeling module is used to use a deep neural network to fuse the video sequence and the depth information to obtain a three-dimensional motion field of the scene, calculate the topological structure change of the point cloud data to generate a deformation vector field, estimate the camera motion trajectory based on the fusion of the acceleration sequence and the angular velocity sequence, input the three-dimensional motion field, the deformation vector field and the motion trajectory into a graph neural network to predict the dynamic area of ​​the scene, apply the modeling algorithm and the parameter combination to the dynamic area for adaptive mesh subdivision and geometric constraint reconstruction, calculate a dynamic threshold according to the scene semantic type, and when the reconstruction error of the dynamic area exceeds the dynamic threshold, reselect the modeling algorithm in the strategy network to reconstruct the dynamic area.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for dynamic modeling of a scene space three-dimensional model based on multimodal data as described in any one of claims 1 to 7 are implemented.

Images