Adaptive modeling method of scene space model based on multi-source data fusion

Abstract

The embodiment of the present application provides a method for adaptive modeling of scene space model by fusion of multi-source data, which constructs complete scene information by fusing video sequences, point cloud data and IMU data. A deep learning network is used for scene analysis, and the scene complexity score is calculated based on the scale coefficient, occlusion degree and geometric deformation coefficient to achieve intelligent matching of modeling strategies. At the same time, the dynamic area of ​​the scene is predicted by using the optical flow algorithm, geometric deformation field and camera pose change, and the modeling quality is monitored by reconstruction error to achieve adaptive switching of the modeling algorithm. This method effectively solves the problems of unstable accuracy and poor adaptability of traditional modeling solutions in complex dynamic scenes, and improves the quality and efficiency of three-dimensional scene reconstruction.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a scene space model adaptive modeling method of multi-source data fusion. Background Art

[0002] With the development of 3D reconstruction technology, scene space modeling is increasingly used in smart cities, indoor navigation and other fields. Traditional scene modeling methods often rely on a single data source, such as using only visual images or laser point cloud data, which makes the modeling process vulnerable to factors such as occlusion and lighting changes, making it difficult to ensure the integrity and accuracy of the model. Although there are some modeling solutions for multi-source data fusion, these solutions usually adopt fixed modeling strategies and cannot adaptively adjust the modeling method according to the characteristics of the scene.

[0003] When dealing with complex dynamic scenes, existing systems generally have problems such as low modeling efficiency and unstable accuracy. Especially in complex indoor and outdoor environments, factors such as dynamic targets, lighting changes, and geometric occlusions in the scene will seriously affect the modeling quality. At the same time, existing methods lack an effective evaluation mechanism for scene complexity and cannot intelligently select the optimal modeling strategy based on scene characteristics, which leads to a large gap between modeling results and actual application needs. In addition, the system's ability to adapt to dynamic scene changes is insufficient, and it is difficult to adjust modeling parameters in time to respond to scene changes.

[0004] Therefore, how to achieve effective fusion of multi-source data, establish a scene complexity assessment mechanism, and improve the system's adaptability to dynamic scenes are key technical issues that need to be solved urgently. This is not only related to the quality and efficiency of 3D reconstruction, but also an important direction for promoting the development of spatial information technology. Summary of the invention

[0005] In response to the problems in the prior art, the present application provides a scene space model adaptive modeling method based on multi-source data fusion, which can effectively solve the problems of unstable accuracy and poor adaptability of traditional modeling solutions in complex dynamic scenes, and improve the quality and efficiency of three-dimensional scene reconstruction.

[0006] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a scene space model adaptive modeling method of multi-source data fusion, comprising:

[0008] Collect video sequences, point cloud data and IMU data of indoor and outdoor scenes, perform noise filtering and geometric correction on the data, extract SIFT image features from the video sequence, extract three-dimensional geometric features from the point cloud data, extract motion posture features from the IMU data, establish a corresponding relationship between the features, and use an adaptive weighted Kalman filter algorithm to fuse the features to obtain multi-source scene information;

[0009] Constructing a deep learning scene analysis network, performing semantic segmentation on the multi-source scene information to obtain the scene type, voxelizing the point cloud data to calculate the scene scale coefficient, using a ray projection method to count the viewpoint visibility of the multi-source scene information to obtain the occlusion degree coefficient, constructing a local curvature histogram based on the three-dimensional geometric features to calculate the geometric deformation coefficient, performing weighted summation on the scale coefficient, the occlusion degree coefficient and the geometric deformation coefficient to obtain a scene complexity score, constructing a feature vector according to the scene type and the complexity score, and retrieving the best matching modeling algorithm and parameter combination in a modeling strategy database;

[0010] The optical flow algorithm is used to track feature points in the video sequence to obtain motion vectors, the inter-frame differences of the point cloud data are calculated to generate a geometric deformation field, the six-degree-of-freedom posture change of the camera is estimated based on the IMU data, the motion vector, the geometric deformation field and the posture change are input into a recursive neural network to predict the dynamic area of ​​the scene, the reconstruction error is calculated for the modeling result of the dynamic area and the multi-source scene information, and when the reconstruction error exceeds an adaptive threshold, the modeling algorithm is switched to rebuild the scene model.

[0011] Furthermore, the video sequence, point cloud data and IMU data of indoor and outdoor scenes are collected, noise is filtered and geometric correction is performed on the data, SIFT image features are extracted from the video sequence, three-dimensional geometric features are extracted from the point cloud data, and motion posture features are extracted from the IMU data, including:

[0012] Performing image noise reduction on the collected video sequence using Gaussian filtering, filtering noise points on the point cloud data using bilateral filtering, smoothing the signal of the IMU data using wavelet transform, establishing a unified time series index for the data, and performing time stamp alignment and geometric correction on the video sequence, the point cloud data, and the IMU data;

[0013] The SIFT feature points and feature descriptors are extracted from the video sequence using a scale-invariant feature transformation algorithm, the normal vector and local curvature features of the point cloud data are calculated to obtain three-dimensional geometric features, the acceleration and angular velocity of the IMU data are obtained through attitude calculation, and the motion attitude features are estimated using Kalman filtering.

[0014] Furthermore, the establishing of the correspondence between the features and the use of an adaptive weighted Kalman filter algorithm to fuse the features to obtain multi-source scene information includes:

[0015] Calculate the matching degree of feature points based on the SIFT feature descriptor, remove wrong matching points using the RANSAC algorithm, project the three-dimensional geometric features onto the image plane to establish a spatial correspondence with the SIFT feature points, and perform spatiotemporal registration of the motion posture features with the SIFT feature points and the three-dimensional geometric features;

[0016] The weight coefficient of the SIFT feature is calculated according to the matching degree of the feature points, the weight coefficient is determined based on the local curvature value of the three-dimensional geometric feature, the weight coefficient is set using the estimation error of the motion posture feature, the weight coefficient is input into the Kalman filter to construct the state estimation equation, and the features are weightedly fused to obtain multi-source scene information.

[0017] Furthermore, the deep learning scene analysis network is constructed, semantic segmentation is performed on the multi-source scene information to obtain the scene type, voxelization is performed on the point cloud data to calculate the scene scale coefficient, the viewpoint visibility of the multi-source scene information is counted using a ray projection method to obtain an occlusion degree coefficient, and a local curvature histogram is constructed based on the three-dimensional geometric features to calculate a geometric deformation coefficient, including:

[0018] Constructing a U-Net convolutional neural network including an encoder and a decoder, wherein the encoder uses a residual structure to extract multi-scale features of the multi-source scene information, and the decoder restores feature resolution through upsampling and skip connections, performs pixel-level semantic segmentation on the multi-source scene information, and uses a fully connected layer to map the segmentation results to scene type labels;

[0019] A three-dimensional grid is established for the point cloud data to divide voxels, the number of points in each voxel is counted to calculate the space occupancy rate to obtain the scene scale coefficient, sampling rays are projected to the scene from different viewpoints, the number of intersections between the rays and the multi-source scene information is calculated to obtain the occlusion degree coefficient, a local curvature histogram is constructed for the normal vectors and principal curvature values ​​in the three-dimensional geometric features, and the geometric deformation coefficient is calculated through the histogram peak distribution.

[0020] Further, the scale coefficient, the occlusion degree coefficient and the geometric deformation coefficient are weightedly summed to obtain a scene complexity score, a feature vector is constructed according to the scene type and the complexity score, and the best matching modeling algorithm and parameter combination are retrieved from the modeling strategy database, including:

[0021] A normalized weight coefficient w1 is set according to the numerical range of the scale coefficient, a normalized weight coefficient w2 is determined based on the distribution characteristics of the occlusion degree coefficient, and a normalized weight coefficient w3 is determined according to the statistical characteristics of the geometric deformation coefficient, and a weighted sum operation is performed on the coefficients to obtain a scene complexity score value between 0 and 1;

[0022] The scene type is converted into a type feature by one-hot encoding, the scene complexity score is converted into a numerical feature, the type feature and the numerical feature are connected to construct a high-dimensional feature vector, and the k-nearest neighbor algorithm is used to retrieve the strategy template most similar to the feature vector in a pre-built modeling strategy database to extract the corresponding modeling algorithm and parameter combination.

[0023] Furthermore, the method of using an optical flow algorithm to track feature points in the video sequence to obtain motion vectors, calculating the inter-frame differences of the point cloud data to generate a geometric deformation field, estimating the six-degree-of-freedom posture change of the camera based on the IMU data, and inputting the motion vectors, the geometric deformation field and the posture change into a recursive neural network to predict the dynamic area of ​​the scene includes:

[0024] The Lucas-Kanade optical flow algorithm is used to calculate the pixel displacement field of adjacent frames of the video sequence, the corner point detection is used to screen stable feature points, the motion vector of the feature point is calculated, the Euclidean distance of the adjacent frame point set of the point cloud data is calculated to obtain the displacement vector field, the nearest neighbor search is used to establish the point correspondence relationship to generate the geometric deformation field, and the extended Kalman filter is used to perform attitude solution on the IMU data to obtain the camera's translation vector and rotation quaternion;

[0025] A long short-term memory network LSTM is constructed to model the temporal features. The feature sequence composed of the motion vector is input into the first LSTM unit, the feature sequence composed of the geometric deformation field is input into the second LSTM unit, and the feature sequence composed of the translation vector and rotation quaternion is input into the third LSTM unit. The output features of the three LSTM units are fused and a probability map of the dynamic area of ​​the scene is generated through a fully connected layer.

[0026] Furthermore, the modeling result for the dynamic area and the multi-source scene information calculate a reconstruction error, and when the reconstruction error exceeds an adaptive threshold, the modeling algorithm is switched to reconstruct the scene model, including:

[0027] The probability map of the dynamic area of ​​the scene is binarized to obtain a dynamic area mask, the modeling result is projected onto the image plane to obtain a depth map, the root mean square error and point cloud registration error between the depth map and the multi-source scene information in the dynamic area are calculated, and the error is slidingly counted using the exponentially weighted average method to obtain the reconstruction error

[0028] An adaptive threshold is calculated based on historical data of the reconstruction error, and a dual threshold mechanism is used for error judgment. When the reconstruction error is greater than a high threshold or is greater than a low threshold for multiple consecutive frames, the modeling algorithm is re-retrieved in the modeling strategy database, and the retrieved new modeling algorithm is applied to scene reconstruction in dynamic areas.

[0029] In a second aspect, the present application provides a scene space model adaptive modeling device for multi-source data fusion, comprising:

[0030] A multi-source scene determination module is used to collect video sequences, point cloud data and IMU data of indoor and outdoor scenes, perform noise filtering and geometric correction on the data, extract SIFT image features from the video sequence, extract three-dimensional geometric features from the point cloud data, extract motion posture features from the IMU data, establish a corresponding relationship between the features, and use an adaptive weighted Kalman filter algorithm to fuse the features to obtain multi-source scene information;

[0031] A scene analysis module is used to construct a deep learning scene analysis network, perform semantic segmentation on the multi-source scene information to obtain the scene type, voxelize the point cloud data to calculate the scene scale coefficient, use a ray projection method to count the viewpoint visibility of the multi-source scene information to obtain the occlusion degree coefficient, construct a local curvature histogram based on the three-dimensional geometric features to calculate the geometric deformation coefficient, perform weighted summation on the scale coefficient, the occlusion degree coefficient and the geometric deformation coefficient to obtain a scene complexity score, construct a feature vector according to the scene type and the complexity score, and retrieve the best matching modeling algorithm and parameter combination in the modeling strategy database;

[0032] The adaptive modeling module is used to track the feature points in the video sequence using an optical flow algorithm to obtain motion vectors, calculate the inter-frame differences of the point cloud data to generate a geometric deformation field, estimate the six-degree-of-freedom posture change of the camera based on the IMU data, input the motion vector, the geometric deformation field and the posture change into a recursive neural network to predict the dynamic area of ​​the scene, calculate the reconstruction error based on the modeling result of the dynamic area and the multi-source scene information, and switch the modeling algorithm to reconstruct the scene model when the reconstruction error exceeds an adaptive threshold.

[0033] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for adaptively modeling a scene space model using multi-source data fusion when executing the program.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for adaptively modeling a scene space model using multi-source data fusion.

[0035] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the method for adaptive modeling of a scene space model using multi-source data fusion.

[0036] It can be seen from the above technical solution that the present application provides a method for adaptive modeling of scene space models with multi-source data fusion, which constructs complete scene information by fusing video sequences, point cloud data and IMU data. A deep learning network is used for scene analysis, and the scene complexity score is calculated based on the scale coefficient, occlusion degree and geometric deformation coefficient to achieve intelligent matching of modeling strategies. At the same time, the dynamic area of ​​the scene is predicted using the optical flow algorithm, geometric deformation field and camera pose changes, and the modeling quality is monitored by reconstruction error to achieve adaptive switching of the modeling algorithm. This method effectively solves the problems of unstable accuracy and poor adaptability of traditional modeling solutions in complex dynamic scenes, and improves the quality and efficiency of three-dimensional scene reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is one of the flow charts of the method for adaptive modeling of scene space model by multi-source data fusion in the embodiment of the present application;

[0039] Figure 2 This is a second flow chart of the method for adaptive modeling of scene space model by multi-source data fusion in an embodiment of the present application;

[0040] Figure 3 This is a third flow chart of the method for adaptive modeling of scene space model by multi-source data fusion in an embodiment of the present application;

[0041] Figure 4 This is a fourth flow chart of the method for adaptive modeling of scene space model by multi-source data fusion in an embodiment of the present application;

[0042] Figure 5 This is a fifth flow chart of the method for adaptive modeling of scene space model by multi-source data fusion in an embodiment of the present application;

[0043] Figure 6 This is a sixth flow chart of the method for adaptive modeling of scene space model by multi-source data fusion in an embodiment of the present application;

[0044] Figure 7 This is a flowchart of the method for adaptive modeling of a scene space model by fusion of multi-source data in an embodiment of the present application, No. 7;

[0045] Figure 8 It is a structural diagram of a scene space model adaptive modeling device for multi-source data fusion in an embodiment of the present application;

[0046] Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0047] Reference numerals:

[0048] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0051] Taking into account the problems existing in the prior art, the present application provides a method for adaptive modeling of scene space models by fusion of multi-source data, which constructs complete scene information by fusing video sequences, point cloud data and IMU data. A deep learning network is used for scene analysis, and the scene complexity score is calculated based on the scale coefficient, occlusion degree and geometric deformation coefficient to achieve intelligent matching of modeling strategies. At the same time, the dynamic area of ​​the scene is predicted by using the optical flow algorithm, geometric deformation field and camera pose changes, and the modeling quality is monitored by reconstruction error to achieve adaptive switching of the modeling algorithm. This method effectively solves the problems of unstable accuracy and poor adaptability of traditional modeling schemes in complex dynamic scenes, and improves the quality and efficiency of three-dimensional scene reconstruction.

[0052] In order to effectively solve the problems of unstable accuracy and poor adaptability of traditional modeling solutions in complex dynamic scenes and improve the quality and efficiency of three-dimensional scene reconstruction, this application provides an embodiment of a scene space model adaptive modeling method based on multi-source data fusion, see Figure 1 The scene space model adaptive modeling method of multi-source data fusion specifically includes the following contents:

[0053] Step S101: collecting video sequences, point cloud data and IMU data of indoor and outdoor scenes, performing noise filtering and geometric correction on the data, extracting SIFT image features from the video sequence, extracting three-dimensional geometric features from the point cloud data, extracting motion posture features from the IMU data, establishing a corresponding relationship between the features, and using an adaptive weighted Kalman filter algorithm to fuse the features to obtain multi-source scene information;

[0054] Optionally, this embodiment proposes a method for collecting, preprocessing and fusing features of multi-source data, and realizes high-precision three-dimensional reconstruction of indoor and outdoor scenes by collaboratively processing video sequences, point cloud data and IMU data.

[0055] This embodiment first adopts a multi-sensor collaborative acquisition strategy, using an RGB-D camera to obtain video sequences and depth information, a lidar to collect high-precision point cloud data, and an IMU sensor to record the motion status of the device. During the data acquisition process, the sampling frequency is dynamically adjusted according to the characteristics of different scenes, focusing on the collection of point cloud details for indoor scenes and increasing the video sampling frequency for outdoor scenes to ensure data quality.

[0056] To improve data quality, this embodiment designs a multi-level data preprocessing process. First, Gaussian filtering is applied to the video sequence to eliminate image noise, and the filter kernel size is adaptively adjusted according to the scene texture complexity. Bilateral filtering is used for point cloud data, while considering spatial distance and normal vector differences to effectively retain edge features. IMU data is transformed through wavelet transformation to remove high-frequency jitter and retain effective motion information.

[0057] In the feature extraction stage, this embodiment designs feature extraction strategies for different data sources. When extracting SIFT features from video sequences, an adaptive scale selection mechanism is used to adjust the feature point detection threshold according to the scene depth information to ensure uniform feature distribution. The three-dimensional geometric feature extraction of point cloud data includes normal vector estimation and local curvature calculation, and an adaptive neighborhood selection method is used to improve feature stability.

[0058] This embodiment innovatively proposes a method for establishing feature correspondence. 2D-3D feature correspondence is established through projection mapping, and the RANSAC algorithm is combined to eliminate mismatches. At the same time, the motion constraints provided by IMU data are used to optimize feature matching results. In complex scenes, temporal consistency is ensured through feature tracking.

[0059] In order to achieve efficient fusion of multi-source data, this embodiment designs an adaptive weighted Kalman filter algorithm. The weight coefficient is dynamically adjusted based on feature reliability, the SIFT feature weight is related to the matching confidence, the geometric feature weight considers the curvature change, and the IMU feature weight is determined based on the signal stability.

[0060] In practical applications, this embodiment is optimized for different scene characteristics. In outdoor scenes with drastic lighting changes, the weight of IMU data is increased; in indoor scenes with rich textures, the weight of visual features is increased; in areas with complex geometric structures, the contribution of point cloud features is strengthened.

[0061] By fusing multi-source data, this embodiment solves the limitations of a single data source in complex scene reconstruction. Video sequences provide rich texture information, point cloud data ensures geometric accuracy, and IMU data supplements motion constraints. The three complement each other and synergize to improve the robustness of scene reconstruction.

[0062] This embodiment performs well in dynamic scene reconstruction. The IMU data is used to assist in judging camera motion and accurately distinguish dynamic objects and static structures in the scene. Multi-source data fusion provides rich scene descriptions, laying the foundation for subsequent adaptive modeling.

[0063] Practice shows that this embodiment can adapt to various complex environments. In indoor scenes with insufficient lighting, point cloud features provide reliable geometric constraints; in fast motion, IMU data ensures the accuracy of posture estimation; when the scene changes dramatically, multi-source feature fusion ensures reconstruction stability. This solution is particularly suitable for application scenarios that require high-precision 3D reconstruction, such as indoor navigation, augmented reality and other fields.

[0064] Step S102: constructing a deep learning scene analysis network, performing semantic segmentation on the multi-source scene information to obtain the scene type, voxelizing the point cloud data to calculate the scene scale coefficient, using a ray projection method to count the viewpoint visibility of the multi-source scene information to obtain the occlusion degree coefficient, constructing a local curvature histogram based on the three-dimensional geometric features to calculate the geometric deformation coefficient, performing weighted summation on the scale coefficient, the occlusion degree coefficient and the geometric deformation coefficient to obtain a scene complexity score, constructing a feature vector according to the scene type and the complexity score, and retrieving the best matching modeling algorithm and parameter combination in the modeling strategy database;

[0065] Optionally, this embodiment proposes a scene analysis and complexity assessment method based on deep learning, which realizes intelligent selection of scene modeling strategies through multi-dimensional feature analysis.

[0066] This embodiment constructs an improved U-Net convolutional neural network architecture, introduces a residual connection mechanism in the encoder, and enhances feature extraction capabilities. The decoder adopts a multi-scale feature fusion strategy to integrate semantic information at different levels through jump connections. The network input includes RGB images, depth maps, and point cloud projection maps, and outputs pixel-level semantic segmentation results. According to the characteristics of different indoor and outdoor scenes, a scene type classification layer is designed to map the semantic segmentation results to specific scene categories.

[0067] In order to accurately assess the scale of the scene, this embodiment uses an adaptive voxelization method to process point cloud data. The voxel size is dynamically adjusted according to the scene scale, and the scale factor is calculated by counting the voxel occupancy rate. In indoor scenes, the voxel size is small to preserve fine structures; in outdoor scenes, the voxel size is appropriately increased to balance computational efficiency and accuracy.

[0068] This embodiment innovatively proposes an occlusion analysis method based on ray projection. By uniformly sampling viewpoints around the scene, projecting rays from each viewpoint to the scene, and counting the number of intersections between the rays and the scene to evaluate the degree of occlusion. According to the occlusion characteristics of different areas, the ray sampling density is adaptively adjusted to improve the accuracy of occlusion evaluation.

[0069] The geometric deformation analysis uses a multi-scale local curvature histogram. This embodiment calculates the principal curvature and Gaussian curvature of the point cloud data and constructs a curvature value distribution histogram. The geometric complexity of the scene is evaluated by analyzing the peak distribution and variance characteristics of the histogram. The number of sampling points is increased in areas with rich details to improve the accuracy of the geometric feature description.

[0070] This embodiment designs an adaptive weight fusion mechanism to comprehensively evaluate the scene complexity by combining the scale factor, occlusion coefficient and geometric deformation coefficient. The weight coefficient is dynamically adjusted according to the scene type. For example, the occlusion coefficient weight is increased in indoor environments, while the scale coefficient influence is strengthened in outdoor open scenes.

[0071] In order to achieve accurate modeling strategy matching, this embodiment constructs a feature vector encoding scheme. The scene type is represented by one-hot encoding, and the complexity score is normalized and concatenated with the type feature. The modeling strategy database stores the optimal modeling parameter combinations under different scene conditions, and the most similar strategy template is retrieved through the k-nearest neighbor algorithm.

[0072] In practical applications, this embodiment shows excellent adaptability. For indoor scenes such as rooms, structural elements such as walls and floors are accurately identified, and a modeling algorithm suitable for plane feature reconstruction is selected. In complex outdoor environments, appropriate geometric modeling strategies are matched according to the distribution characteristics of vegetation and buildings.

[0073] This embodiment effectively solves the problem of parameter selection in traditional modeling methods. It realizes scene semantic understanding through deep learning, combines multi-dimensional complexity evaluation, and automatically selects the optimal modeling strategy. This solution is particularly suitable for applications that need to handle diverse scenes, such as urban modeling, indoor scene reconstruction, etc.

[0074] Practice has proved that this embodiment can accurately evaluate the characteristics of various scenes. In indoor environments with dense furniture, it can accurately calculate occlusion relationships; in building complex scenes, it can accurately evaluate geometric complexity; in dynamic environments, it can quickly adapt to scene changes and adjust modeling strategies. This adaptive modeling solution significantly improves the quality and efficiency of 3D reconstruction.

[0075] Step S103: Using an optical flow algorithm to track feature points in the video sequence to obtain motion vectors, calculating the inter-frame differences of the point cloud data to generate a geometric deformation field, estimating the six-degree-of-freedom posture change of the camera based on the IMU data, inputting the motion vector, the geometric deformation field and the posture change into a recursive neural network to predict the dynamic area of ​​the scene, calculating the reconstruction error based on the modeling result of the dynamic area and the multi-source scene information, and switching the modeling algorithm to rebuild the scene model when the reconstruction error exceeds an adaptive threshold.

[0076] Optionally, this embodiment proposes a dynamic scene analysis and adaptive reconstruction method based on multi-source data, which accurately captures the dynamic characteristics of the scene by fusing visual, geometric and inertial information.

[0077] This embodiment uses an improved pyramid Lucas-Kanade optical flow algorithm to track feature points in video sequences. The algorithm handles large-scale motion through a multi-layer image pyramid structure, and uses an iterative optimization method to calculate the displacement of feature points at each layer. To improve tracking stability, the search window size is dynamically adjusted in combination with image gradient information, and reliable motion vectors are screened using the brightness consistency constraint of the previous and next frames.

[0078] The calculation of the geometric deformation field adopts an innovative point cloud registration method. This embodiment first performs coarse registration on the point clouds of consecutive frames and establishes the initial correspondence through matching of local feature descriptors. Then, a non-rigid deformation model is applied for fine registration, the displacement vector of each point is calculated, and a dense deformation field describing the scene deformation is generated. In areas with occlusion, the deformation information is supplemented by the motion consistency constraints of adjacent points.

[0079] This embodiment processes IMU data based on the extended Kalman filter to achieve six-degree-of-freedom pose estimation of the camera. The camera motion state is predicted by gyroscope and accelerometer data, and observation updates are performed in combination with visual feature matching results. The process noise and observation noise of the filter are dynamically adjusted according to the intensity of the movement to improve the accuracy of pose estimation.

[0080] In order to achieve accurate recognition of dynamic areas, this embodiment designs a time series prediction model based on the long short-term memory network (LSTM). The network input includes feature point motion vectors, local geometric deformations, and camera pose changes. The time series features are extracted through multi-layer recurrent units, and pixel-level dynamic probability maps are output. The network training adopts an online learning strategy to continuously optimize the model parameters according to the characteristics of scene changes.

[0081] This embodiment innovatively proposes an adaptive modeling quality assessment mechanism. By comparing the reconstructed model with multi-source scene information, the geometric error and appearance error are calculated. The geometric error is based on the point cloud distance metric, and the appearance error is assessed by image projection difference. The error threshold is dynamically adjusted according to the scene type and observation conditions to ensure that the reconstruction quality meets the application requirements.

[0082] In practical applications, this embodiment can effectively handle various dynamic scenes. For indoor environments with moving people, the moving area is accurately identified and a highly adaptable modeling algorithm is used; in scenes with deformable objects, a suitable deformation model is selected through geometric deformation analysis; for situations where the camera moves rapidly, IMU data is used to ensure the continuity of pose estimation.

[0083] This embodiment solves the limitations of traditional reconstruction methods in dynamic scenes. It realizes motion analysis through multi-source data fusion, improves the accuracy of dynamic area recognition through deep learning, and adaptively adjusts the modeling strategy to ensure the quality of reconstruction. This solution is particularly suitable for application scenarios that require real-time 3D reconstruction, such as augmented reality, robot navigation, etc.

[0084] Practice has shown that this embodiment performs well in various dynamic environments. In crowded public places, it accurately tracks dynamic targets; in natural scenes, it accurately reconstructs vegetation swaying in the wind; in indoor scene reconstruction, it effectively handles dynamic changes such as furniture movement. By promptly detecting the degradation of reconstruction quality and switching modeling strategies, the stability and accuracy of dynamic scene reconstruction are significantly improved.

[0085] From the above description, it can be seen that the adaptive modeling method of scene space model with multi-source data fusion provided in the embodiment of the present application can construct complete scene information by fusing video sequences, point cloud data and IMU data. A deep learning network is used for scene analysis, and the scene complexity score is calculated based on the scale coefficient, occlusion degree and geometric deformation coefficient to achieve intelligent matching of modeling strategies. At the same time, the dynamic area of ​​the scene is predicted by using the optical flow algorithm, geometric deformation field and camera pose changes, and the modeling quality is monitored by reconstruction error to achieve adaptive switching of the modeling algorithm. This method effectively solves the problems of unstable accuracy and poor adaptability of traditional modeling solutions in complex dynamic scenes, and improves the quality and efficiency of three-dimensional scene reconstruction.

[0086] In one embodiment of the scene space model adaptive modeling method of multi-source data fusion of the present application, see Figure 2 , and can also include the following:

[0087] Step S201: performing image noise reduction on the collected video sequence using Gaussian filtering, filtering noise points on the point cloud data using bilateral filtering, smoothing the IMU data using wavelet transform, establishing a unified time series index for the data, and performing time stamp alignment and geometric correction on the video sequence, the point cloud data, and the IMU data;

[0088] Step S202: Use the scale-invariant feature transformation algorithm to extract SIFT feature points and feature descriptors from the video sequence, calculate the normal vector and local curvature features of the point cloud data to obtain three-dimensional geometric features, obtain acceleration and angular velocity from the IMU data through attitude solution, and use Kalman filtering to estimate motion attitude features.

[0089] Optionally, this embodiment proposes a comprehensive method of multi-source data preprocessing and feature extraction, which lays the foundation for subsequent scene reconstruction through multi-level data optimization and feature analysis.

[0090] This embodiment first designs a special denoising strategy for different data sources. Adaptive Gaussian filtering is used for video sequences, and the filter kernel size is dynamically adjusted according to the local variance of the image, effectively suppressing noise while retaining image edge details. In scenes with insufficient lighting, the filter strength is appropriately increased; in areas with rich textures, the filter degree is reduced to retain detail information.

[0091] The noise reduction of point cloud data adopts an improved bilateral filtering algorithm, taking into account the spatial distance and normal vector difference of the points. This embodiment determines the filtering range through adaptive neighborhood selection, adopts a smaller search radius for dense areas, and expands the search range for sparse areas. This strategy effectively removes outliers and measurement noise while maintaining geometric features.

[0092] The smoothing of IMU data adopts the wavelet transform method. This embodiment uses the db4 wavelet basis function to separate the noise components in the signal through multi-scale decomposition. The high-frequency details are retained in the places where the signal changes drastically, and the low-frequency components are strengthened in the stable area to achieve adaptive smoothing of the signal.

[0093] Data synchronization is the key innovation of this embodiment. By establishing a unified timing index system, data sources with different sampling frequencies are aligned to the same time base. An interpolation algorithm is used to handle the problem of inconsistent sampling frequencies to ensure the temporal continuity of the data. The geometric correction process takes into account the installation errors between sensors and calculates the relative pose transformation matrix through the hand-eye calibration method.

[0094] In the feature extraction stage, this embodiment optimizes the SIFT algorithm. By adjusting the number of layers and sampling interval of the Gaussian difference pyramid, the accuracy of feature point detection is improved. When calculating the feature descriptor, the depth information of the image is considered, and different weights are given to the gradient directions of different depth levels to enhance the distinguishability of the features.

[0095] The geometric feature extraction of point cloud data adopts a multi-scale analysis method. This embodiment first estimates the normal vector of each point and determines the local surface direction through principal component analysis. The curvature feature calculation takes into account the geometric changes at different scales and forms a robust feature description by combining the curvature values ​​of multiple scales.

[0096] The IMU data is processed using an improved attitude solution algorithm. This embodiment uses quaternions to represent the device attitude and estimates the motion state by combining accelerometer and gyroscope data. The state vector of the Kalman filter contains position, velocity and attitude information, and the attitude estimation accuracy is improved by adaptively adjusting process noise.

[0097] This embodiment shows excellent performance in practical applications. In indoor scenes, it can accurately extract structural features such as walls and ground; in dynamic environments, it can stably track the motion of the camera; in complex outdoor scenes, it can reliably detect and describe scene feature points.

[0098] Through preprocessing and feature extraction of multi-source data, this embodiment solves the limitations of a single data source in complex environments. Video sequences provide rich texture information, point cloud data ensures geometric accuracy, and IMU data supplements motion constraints. The three complement each other and synergize to improve the reliability of feature extraction. This comprehensive processing solution is particularly suitable for applications that require precise scene understanding, such as augmented reality, robot navigation, and other fields.

[0099] Practice has proved that this embodiment can adapt to various complex application scenarios. It can maintain the stability of feature detection in an environment with changing lighting, accurately estimate the camera posture during rapid motion, and accurately extract three-dimensional features in areas with complex geometric structures. Through the collaborative processing of multi-source data, the accuracy and robustness of scene reconstruction are significantly improved.

[0100] In one embodiment of the scene space model adaptive modeling method of multi-source data fusion of the present application, see Figure 3 , and can also include the following:

[0101] Step S301: Calculate the matching degree of feature points based on the SIFT feature descriptor, remove wrong matching points using the RANSAC algorithm, project the three-dimensional geometric features onto the image plane to establish a spatial correspondence with the SIFT feature points, and perform spatiotemporal registration of the motion posture features with the SIFT feature points and the three-dimensional geometric features;

[0102] Step S302: Calculate the weight coefficient of the SIFT feature according to the matching degree of the feature points, determine the weight coefficient based on the local curvature value of the three-dimensional geometric feature, set the weight coefficient using the estimated error of the motion posture feature, input the weight coefficient into the Kalman filter to construct a state estimation equation, and perform weighted fusion on the features to obtain multi-source scene information.

[0103] Optionally, this embodiment proposes an innovative method for multi-source feature matching and fusion, which achieves accurate fusion of scene information through feature space mapping and adaptive weight allocation.

[0104] This embodiment first uses an improved feature matching strategy to process SIFT features. The initial matching pair is obtained by calculating the Euclidean distance between feature descriptors, and a ratio test mechanism is introduced to screen reliable matches. For areas with rich textures, a strict ratio threshold is used; while in areas with single textures, the matching conditions are appropriately relaxed. The sampling strategy of the RANSAC algorithm is dynamically adjusted according to the characteristics of the scene, and the number of iterations is increased in areas with repeated textures to improve the accuracy of outlier removal.

[0105] The correspondence between 3D geometric features and image features is established using a projection mapping method. This embodiment first uses camera calibration parameters to project point cloud data onto the image plane, and establishes spatial correspondence through local feature descriptor matching. To handle projection ambiguity, depth consistency constraints and local structure similarity verification are introduced to ensure the accuracy of feature correspondence.

[0106] The spatiotemporal registration process innovatively adopts a multi-level registration strategy. This embodiment first uses the rough pose provided by the IMU for initial alignment, and then optimizes the relative pose transformation through feature matching. In dynamic scenes, the spatiotemporal transformation parameters are updated in real time to ensure the synchronization of different data sources.

[0107] The feature weight allocation is the core innovation of this embodiment. SIFT feature weights are calculated based on matching confidence, taking into account the scale, directional consistency and descriptor distance of feature points. Recurring stable features are given higher weights, while temporary features are reduced in their impact.

[0108] The weight calculation of three-dimensional geometric features comprehensively considers the significance and stability of curvature features. This embodiment reduces the weight of geometric features in flat areas, and increases the weight value in areas with obvious features such as edges and corners. By analyzing the local curvature distribution, the weight coefficient is adaptively adjusted to improve the accuracy of feature fusion.

[0109] The weight setting of motion posture features is based on the state estimation error of the Kalman filter. This embodiment analyzes the covariance of acceleration and angular velocity measurements to evaluate the reliability of posture estimation. When the IMU data is stable, its weight is increased, and the weight is appropriately reduced in the case of intense movement or magnetic field interference.

[0110] Feature fusion uses an improved Kalman filter framework. This embodiment designs an extended state vector, which includes position, attitude and feature parameters. The observation equation integrates visual, geometric and inertial information, and optimizes state estimation performance by adaptively adjusting process noise and observation noise.

[0111] In practical applications, this embodiment shows excellent adaptability. In indoor scene reconstruction, the geometric features and visual features of walls and ground are accurately integrated; in dynamic environments, the camera motion state is stably estimated; in complex outdoor scenes, multi-source information is reliably integrated.

[0112] This embodiment effectively solves the problem of traditional feature fusion methods in complex scenes. Through multi-feature collaboration and adaptive weight allocation, the integrity and accuracy of scene information are improved. This solution is particularly suitable for applications that require high-precision 3D reconstruction, such as augmented reality, robot navigation, etc.

[0113] Practice has proved that this embodiment can adapt to various application environments. It can maintain the stability of feature matching in scenes with changing lighting, accurately fuse multi-source data during rapid motion, and accurately reconstruct scene details in areas with complex geometric structures. Through intelligent fusion of features, the quality and reliability of scene reconstruction are significantly improved.

[0114] In one embodiment of the scene space model adaptive modeling method of multi-source data fusion of the present application, see Figure 4, and can also include the following:

[0115] Step S401: constructing a U-Net convolutional neural network including an encoder and a decoder, wherein the encoder uses a residual structure to extract multi-scale features of the multi-source scene information, and the decoder restores feature resolution through upsampling and skip connections, performs pixel-level semantic segmentation on the multi-source scene information, and uses a fully connected layer to map the segmentation results to scene type labels;

[0116] Step S402: Establish a three-dimensional grid for the point cloud data and divide it into voxels, count the number of points in each voxel and calculate the space occupancy rate to obtain the scene scale coefficient, project sampling rays to the scene from different viewpoints, calculate the number of intersections between the rays and the multi-source scene information to obtain the occlusion degree coefficient, construct a local curvature histogram for the normal vectors and principal curvature values ​​in the three-dimensional geometric features, and calculate the geometric deformation coefficient through the histogram peak distribution.

[0117] Optionally, this embodiment proposes a scene segmentation and feature analysis method based on deep learning, which realizes accurate scene understanding and geometric feature extraction through neural networks and spatial feature statistics.

[0118] This embodiment designs an improved U-Net network structure, and the encoder uses deep residual blocks to improve feature extraction capabilities. Each residual block contains two convolutional layers and a batch normalization layer, and the gradient vanishing problem is alleviated by short-circuiting connections. When processing complex scenes, the deeper network layer can capture global semantic information, and the shallow network retains local detail features.

[0119] The decoder uses transposed convolution to achieve feature map upsampling, and introduces a skip connection mechanism to fuse multiple layers of features. This embodiment uses an attention mechanism when fusing features, and adaptively adjusts the fusion weights according to the importance of the features. This design effectively improves the accuracy of the segmentation boundary while retaining the details.

[0120] The design of the fully connected layer takes into account the diversity of scenes, and uses a multi-branch structure to process different types of scene features. This embodiment uses a dropout layer to prevent overfitting, and uses a softmax activation function in the output layer to obtain the probability distribution of scene types.

[0121] The three-dimensional spatial feature analysis adopts an adaptive voxel division strategy. This embodiment dynamically adjusts the voxel size according to the point cloud density, using smaller voxels in the object boundary area to improve accuracy, and using larger voxels in the flat area to improve calculation efficiency. The calculation of spatial occupancy takes into account the distribution characteristics of the points and reflects the spatial complexity of the scene through weighted statistics.

[0122] The occlusion analysis innovatively adopts a multi-viewpoint ray tracing method. This embodiment designs a uniformly distributed viewpoint sampling strategy to emit rays from different angles to detect the scene structure. The number of intersections between the ray and the scene reflects the complexity of the space, and the occlusion degree coefficient is obtained through statistical analysis.

[0123] The geometric feature analysis uses a multi-scale feature extraction method. This embodiment first calculates the normal vector and principal curvature of the local surface, and determines the feature calculation range through a spherical neighborhood search. The construction of the curvature histogram takes directionality into account, and divides the space into multiple directional intervals to statistically calculate the curvature distribution.

[0124] The calculation of deformation coefficient innovatively introduces histogram peak analysis. This embodiment evaluates the degree of change of geometric structure by detecting the main peak position and amplitude of the histogram. Multiple peaks represent complex geometric structures, while a single peak corresponds to a simple plane or sphere.

[0125] In practical applications, this embodiment demonstrates excellent scene understanding capabilities. In indoor environments, it accurately identifies different types of objects such as walls, floors, and furniture; in outdoor scenes, it reliably segments structures such as buildings, vegetation, and roads; and in complex environments, it accurately analyzes spatial structures and geometric features.

[0126] This embodiment solves the limitations of traditional scene analysis methods in complex environments. By combining deep learning and geometric feature analysis, the accuracy and robustness of scene understanding are improved. This solution is particularly suitable for applications that require accurate scene understanding, such as robot navigation, augmented reality, etc.

[0127] Practice has proved that this embodiment can adapt to various application scenarios. It can maintain the stability of segmentation results in an environment with changing lighting, accurately analyze spatial structures in densely occluded areas, and accurately extract features in complex geometric areas. Through the synergy of deep learning and geometric analysis, the quality and reliability of scene understanding are significantly improved.

[0128] In one embodiment of the scene space model adaptive modeling method of multi-source data fusion of the present application, see Figure 5 , and can also include the following:

[0129] Step S501: setting a normalized weight coefficient w1 according to the numerical range of the scale coefficient, determining a normalized weight coefficient w2 based on the distribution characteristics of the occlusion degree coefficient, and determining a normalized weight coefficient w3 according to the statistical characteristics of the geometric deformation coefficient, and performing a weighted sum operation on the coefficients to obtain a scene complexity score value between 0 and 1;

[0130] Step S502: Convert the scene type into a type feature by one-hot encoding, convert the scene complexity score into a numerical feature, connect the type feature and the numerical feature to construct a high-dimensional feature vector, use the k-nearest neighbor algorithm to retrieve the strategy template most similar to the feature vector in a pre-built modeling strategy database, and extract the corresponding modeling algorithm and parameter combination.

[0131] Optionally, this embodiment proposes a scene complexity assessment and modeling strategy selection method based on multi-dimensional features, and realizes intelligent scene modeling through adaptive weight allocation and feature matching.

[0132] This embodiment first assigns weights to the scale factor. By analyzing the relationship between the scale of the scene and the difficulty of modeling, a piecewise linear mapping function is designed. When the scale factor is in the medium range, the weight factor w1 changes slowly; in the maximum or minimum value range, the weight factor grows rapidly, reflecting the difficulty of modeling abnormal scale scenes.

[0133] The weight distribution of the occlusion degree coefficient adopts an adaptive method based on statistical distribution. This embodiment analyzes the probability distribution characteristics of the occlusion coefficient and obtains the distribution curve through kernel density estimation. The weight coefficient w2 is increased in the area with severe occlusion, reflecting the significant impact of occlusion on the difficulty of modeling.

[0134] The weight determination of geometric deformation coefficient innovatively introduces statistical feature analysis. This embodiment calculates the mean, variance and other statistics of the deformation coefficient and designs a nonlinear mapping function. When the geometry changes drastically, the weight coefficient w3 increases accordingly, highlighting the impact of geometric complexity on modeling.

[0135] The calculation of the scene complexity score adopts a weighted summation method to ensure that the final score falls between 0 and 1. This embodiment ensures the dimensional consistency of different coefficients through normalization processing, and the weight ratio reflects the contribution of each factor to the modeling difficulty.

[0136] The unique hot encoding of the type feature adopts an optimized encoding scheme. This embodiment designs the encoding mode according to the similarity of the scene type. The encodings of similar types have more identical bits, which is convenient for subsequent similarity calculation. At the same time, considering that the scene may have multiple attributes, soft coding is allowed to represent the uncertainty of the type.

[0137] The construction of feature vectors adopts a multimodal feature fusion strategy. This embodiment effectively combines discrete type features and continuous complexity features, designs an adaptive feature standardization method, and ensures that the weights of different features in similarity calculation are balanced.

[0138] The implementation of the k-nearest neighbor algorithm uses an improved similarity metric. This embodiment introduces feature weights based on the Euclidean distance, and sets different influencing factors for type features and numerical features. By dynamically adjusting the k value, the modeling requirements of different scenarios can be met.

[0139] The construction of the modeling strategy database is based on a lot of practical experience. This embodiment collects modeling cases of different types and different complexity scenarios, extracts key parameters and performance indicators of the modeling algorithm, constructs a strategy template through cluster analysis, and establishes a mapping relationship from feature vectors to modeling strategies.

[0140] In practical applications, this embodiment shows excellent adaptability. In indoor scenes, it accurately evaluates the complexity of furniture placement; in urban environments, it reasonably analyzes the spatial structural characteristics of building complexes; in natural scenes, it reliably evaluates the geometric complexity of terrain.

[0141] This embodiment solves the problem of difficult parameter selection in traditional modeling methods. Through feature analysis and strategy matching, intelligent configuration of modeling parameters is achieved. This solution is particularly suitable for applications that require automated 3D reconstruction, such as urban planning, scene reconstruction, etc.

[0142] Practice has proved that this embodiment can adapt to various modeling scenarios. In simple environments, efficient modeling strategies are selected, in complex scenes, sophisticated reconstruction methods are adopted, and special processing algorithms are used in special structural areas. Through intelligent strategy selection, the efficiency and quality of scene reconstruction are significantly improved.

[0143] In one embodiment of the scene space model adaptive modeling method of multi-source data fusion of the present application, see Figure 6 , and can also include the following:

[0144] Step S601: using the Lucas-Kanade optical flow algorithm to calculate the pixel displacement field of adjacent frames of the video sequence, using corner detection to screen stable feature points, calculating the motion vector of the feature points, calculating the Euclidean distance of adjacent frame point sets for the point cloud data to obtain the displacement vector field, using the nearest neighbor search to establish point correspondence to generate a geometric deformation field, and performing attitude solution on the IMU data through an extended Kalman filter to obtain the camera's translation vector and rotation quaternion;

[0145] Step S602: Construct a long short-term memory network LSTM to model the temporal features, input the feature sequence composed of the motion vector into the first LSTM unit, input the feature sequence composed of the geometric deformation field into the second LSTM unit, and input the feature sequence composed of the translation vector and the rotation quaternion into the third LSTM unit, and fuse the output features of the three LSTM units through a fully connected layer to generate a probability map of the dynamic area of ​​the scene.

[0146] Optionally, this embodiment proposes a scene dynamic feature extraction method that fuses multi-source data, and achieves accurate recognition of scene motion areas by combining optical flow analysis, geometric deformation detection and inertial navigation.

[0147] This embodiment uses an improved Lucas-Kanade algorithm in optical flow calculation. Large motion is handled through a pyramid layering strategy, and iterative optimization is used at each layer to improve accuracy. Corner point detection uses the Harris operator, combined with non-maximum suppression to select significant feature points. The motion vector calculation of the feature point takes into account the brightness consistency and spatial continuity constraints.

[0148] The point cloud data processing innovatively adopts an adaptive search strategy. This embodiment accelerates the nearest neighbor search through octree space partitioning and dynamically adjusts the search radius to adapt to the change of point cloud density. The calculation of the displacement vector field adopts a robust statistical method to eliminate the influence of abnormal point pairs.

[0149] The geometric deformation field is generated by using an improved ICP registration algorithm. This embodiment enhances the accuracy of point correspondence through local feature descriptors and estimates rigid transformation parameters using weighted least squares. For non-rigid deformation, an elastic deformation model is introduced to describe local deformation features.

[0150] IMU data processing uses an extended Kalman filter for attitude solution. This embodiment designs an adaptive noise model to dynamically adjust process noise and observation noise according to the motion state. The quaternion representation avoids the universal lock problem of Euler angles and improves the stability of attitude estimation.

[0151] The temporal feature modeling adopts a multi-branch LSTM network structure. This embodiment designs adaptive LSTM unit structures according to the characteristics of different data sources. The first LSTM unit processes the optical flow features and uses a bidirectional structure to capture the correlation between the previous and next frames. The second LSTM unit analyzes geometric deformation and uses an attention mechanism to highlight changes in important areas. The third LSTM unit integrates motion information and controls the impact of information at different times through a gating mechanism.

[0152] Feature fusion innovatively adopts an adaptive weight mechanism. This embodiment obtains the importance weights of the output features of each LSTM unit through learning, and dynamically adjusts the contribution of different features according to the characteristics of the scene. The fully connected layer adopts a multi-scale structure to extract high-level semantic features layer by layer.

[0153] The probability map is generated using a soft threshold strategy. This embodiment maps features to a probability space using a sigmoid activation function to generate a probability value for each pixel belonging to a dynamic region. In post-processing, conditional random fields are used to optimize spatial consistency and improve the coherence of the probability map.

[0154] In practical applications, this embodiment demonstrates excellent dynamic feature extraction capabilities. In indoor scenes, it accurately identifies the movement of people and the moving areas of objects; in road environments, it reliably detects the movement trajectory of vehicles; and in complex scenes, it accurately analyzes the movement status of multiple targets.

[0155] This embodiment solves the problem of poor robustness of traditional dynamic feature extraction methods. Through multi-source data fusion and time series modeling, the accuracy and reliability of dynamic area detection are improved. This solution is particularly suitable for applications that require precise motion analysis, such as intelligent monitoring, robot navigation, etc.

[0156] Practice has proved that this embodiment can adapt to various dynamic scenes. It can maintain the stability of detection in an environment with changing lighting, accurately track targets in fast-moving areas, and reliably distinguish different motion modes in multi-target scenes. Through the collaborative analysis of multi-source information, the performance of dynamic feature extraction is significantly improved.

[0157] In one embodiment of the scene space model adaptive modeling method of multi-source data fusion of the present application, see Figure 7 , and can also include the following:

[0158] Step S701: Binarize the probability map of the scene dynamic area to obtain a dynamic area mask, project the modeling result onto the image plane to obtain a depth map, calculate the root mean square error and point cloud registration error between the depth map and the multi-source scene information in the dynamic area, and use the exponential weighted average method to perform sliding statistics on the error to obtain a reconstruction error

[0159] Step S702: Calculate an adaptive threshold based on the historical data of the reconstruction error, and use a dual threshold mechanism to make error judgments. When the reconstruction error is greater than a high threshold or is greater than a low threshold for multiple consecutive frames, re-retrieve the modeling algorithm in the modeling strategy database, and apply the retrieved new modeling algorithm to scene reconstruction in the dynamic area.

[0160] Optionally, this embodiment proposes an adaptive scene reconstruction method based on error detection, which achieves accurate reconstruction of the scene by dynamically evaluating the reconstruction quality and adjusting the modeling strategy in a timely manner.

[0161] This embodiment uses an improved binarization algorithm to process the dynamic region probability map. Adaptive threshold segmentation is combined with morphological operations to optimize the mask boundary to ensure the integrity of the dynamic region. For the boundary area where the probability value is close to the threshold, gradient processing is used to avoid artifacts caused by hard segmentation.

[0162] The depth map is generated using an accurate projection model. This embodiment considers the internal and external parameters and distortion characteristics of the camera and maps the 3D point cloud to the image plane through the reprojection matrix. When processing the occluded area, the z-buffer algorithm is used to ensure the correctness of the depth value.

[0163] The error calculation innovatively integrates multiple evaluation indicators. The root mean square error reflects the absolute difference in depth values, and the reliability of pixels is considered through weighted calculation. The point cloud registration error uses the point-to-surface distance metric to better express the degree of fit of the geometric structure.

[0164] Sliding statistics use the exponential weighted average method to give greater weight to recent data. This embodiment balances the stability of historical information and the responsiveness of the current state by adjusting the attenuation factor. The statistical process considers the continuity of scene changes to avoid interference from occasional errors.

[0165] The calculation of the adaptive threshold is based on the error distribution characteristics. This embodiment analyzes the statistical characteristics of historical error data and obtains the error distribution model through kernel density estimation. The threshold setting takes into account the complexity of the scene, adopts a stricter standard in simple scenes, and appropriately relaxes the restrictions in complex scenes.

[0166] The design of the dual threshold mechanism provides robust error judgment. The high threshold detects significant quality degradation and triggers reconstruction strategy adjustment in a timely manner; the low threshold monitors continuous performance degradation to prevent reconstruction failure caused by cumulative errors. A buffer is set between the thresholds to avoid frequent switching of strategies.

[0167] The reconstruction strategy is updated using an intelligent retrieval mechanism. This embodiment analyzes the error pattern and identifies the main reasons for the quality degradation. The retrieval process considers the current scene characteristics and dynamic characteristics and selects the most suitable modeling algorithm and parameter configuration.

[0168] The new strategy application adopts a smooth transition scheme. This embodiment avoids sudden changes in reconstruction results caused by strategy switching by gradually adjusting parameters. For dynamic objects spanning multiple modeling cycles, the temporal continuity of the reconstruction results is ensured.

[0169] In practical applications, this embodiment shows excellent adaptability. In indoor scenes, the geometric structure of moving objects can be accurately reconstructed; in road environments, the morphological features of moving vehicles can be reliably reconstructed; in complex scenes, multiple dynamic targets can be accurately reconstructed.

[0170] This embodiment solves the problem of instability of traditional reconstruction methods in dynamic scenes. Through quality monitoring and strategy adjustment, the robustness and accuracy of scene reconstruction are improved. This solution is particularly suitable for applications that require high-quality 3D reconstruction, such as virtual reality, scene understanding, etc.

[0171] Practice has proved that this embodiment can adapt to various reconstruction scenarios. It can maintain reconstruction quality in environments with changing lighting, accurately capture geometric details in fast-moving areas, and reliably reconstruct spatial structures in complex dynamic scenes. Through adaptive strategy adjustment, the performance of dynamic scene reconstruction is significantly improved.

[0172] In order to effectively solve the problems of unstable accuracy and poor adaptability of traditional modeling solutions in complex dynamic scenes and improve the quality and efficiency of 3D scene reconstruction, the present application provides an embodiment of a scene space model adaptive modeling device for multi-source data fusion for realizing all or part of the contents of the scene space model adaptive modeling method for multi-source data fusion, see Figure 8 The scene space model adaptive modeling device of multi-source data fusion specifically includes the following contents:

[0173] The multi-source scene determination module 10 is used to collect video sequences, point cloud data and IMU data of indoor and outdoor scenes, perform noise filtering and geometric correction on the data, extract SIFT image features from the video sequence, extract three-dimensional geometric features from the point cloud data, extract motion posture features from the IMU data, establish a corresponding relationship between the features, and use an adaptive weighted Kalman filter algorithm to fuse the features to obtain multi-source scene information;

[0174] A scene analysis module 20 is used to construct a deep learning scene analysis network, perform semantic segmentation on the multi-source scene information to obtain a scene type, voxelize the point cloud data to calculate a scene scale coefficient, use a ray projection method to count the viewpoint visibility of the multi-source scene information to obtain an occlusion degree coefficient, construct a local curvature histogram based on the three-dimensional geometric features to calculate a geometric deformation coefficient, perform weighted summation on the scale coefficient, the occlusion degree coefficient and the geometric deformation coefficient to obtain a scene complexity score, construct a feature vector according to the scene type and the complexity score, and retrieve the best matching modeling algorithm and parameter combination in a modeling strategy database;

[0175] The adaptive modeling module 30 is used to track the feature points in the video sequence using an optical flow algorithm to obtain motion vectors, calculate the inter-frame differences of the point cloud data to generate a geometric deformation field, estimate the six-degree-of-freedom posture change of the camera based on the IMU data, input the motion vector, the geometric deformation field and the posture change into a recursive neural network to predict the dynamic area of ​​the scene, calculate the reconstruction error based on the modeling result of the dynamic area and the multi-source scene information, and switch the modeling algorithm to rebuild the scene model when the reconstruction error exceeds an adaptive threshold.

[0176] From the above description, it can be seen that the scene space model adaptive modeling device of multi-source data fusion provided in the embodiment of the present application can construct complete scene information by fusing video sequences, point cloud data and IMU data. A deep learning network is used for scene analysis, and the scene complexity score is calculated based on the scale coefficient, occlusion degree and geometric deformation coefficient to achieve intelligent matching of modeling strategies. At the same time, the dynamic area of ​​the scene is predicted by using the optical flow algorithm, geometric deformation field and camera pose changes, and the modeling quality is monitored by reconstruction error to achieve adaptive switching of the modeling algorithm. This method effectively solves the problems of unstable accuracy and poor adaptability of traditional modeling solutions in complex dynamic scenes, and improves the quality and efficiency of three-dimensional scene reconstruction.

[0177] From the hardware level, in order to effectively solve the problems of unstable accuracy and poor adaptability of traditional modeling solutions in complex dynamic scenes and improve the quality and efficiency of three-dimensional scene reconstruction, the present application provides an embodiment of an electronic device for implementing all or part of the contents of the scene space model adaptive modeling method of multi-source data fusion, and the electronic device specifically includes the following contents:

[0178] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the scene space model adaptive modeling device of multi-source data fusion and the core business system, user terminal and related database and other related devices; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., and the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the scene space model adaptive modeling method of multi-source data fusion and the embodiment of the scene space model adaptive modeling device of multi-source data fusion in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0179] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0180] In practical applications, part of the method for adaptive modeling of scene space model of multi-source data fusion can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0181] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0182] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0183] In one embodiment, the scene space model adaptive modeling method function of multi-source data fusion can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0184] Step S101: collecting video sequences, point cloud data and IMU data of indoor and outdoor scenes, performing noise filtering and geometric correction on the data, extracting SIFT image features from the video sequence, extracting three-dimensional geometric features from the point cloud data, extracting motion posture features from the IMU data, establishing a corresponding relationship between the features, and using an adaptive weighted Kalman filter algorithm to fuse the features to obtain multi-source scene information;

[0185] Step S102: constructing a deep learning scene analysis network, performing semantic segmentation on the multi-source scene information to obtain the scene type, voxelizing the point cloud data to calculate the scene scale coefficient, using a ray projection method to count the viewpoint visibility of the multi-source scene information to obtain the occlusion degree coefficient, constructing a local curvature histogram based on the three-dimensional geometric features to calculate the geometric deformation coefficient, performing weighted summation on the scale coefficient, the occlusion degree coefficient and the geometric deformation coefficient to obtain a scene complexity score, constructing a feature vector according to the scene type and the complexity score, and retrieving the best matching modeling algorithm and parameter combination in the modeling strategy database;

[0186] Step S103: Using an optical flow algorithm to track feature points in the video sequence to obtain motion vectors, calculating the inter-frame differences of the point cloud data to generate a geometric deformation field, estimating the six-degree-of-freedom posture change of the camera based on the IMU data, inputting the motion vector, the geometric deformation field and the posture change into a recursive neural network to predict the dynamic area of ​​the scene, calculating the reconstruction error based on the modeling result of the dynamic area and the multi-source scene information, and switching the modeling algorithm to rebuild the scene model when the reconstruction error exceeds an adaptive threshold.

[0187] From the above description, it can be seen that the electronic device provided in the embodiment of the present application constructs complete scene information by fusing video sequences, point cloud data and IMU data. A deep learning network is used for scene analysis, and the scene complexity score is calculated based on the scale coefficient, occlusion degree and geometric deformation coefficient to achieve intelligent matching of modeling strategies. At the same time, the dynamic area of ​​the scene is predicted using the optical flow algorithm, geometric deformation field and camera pose changes, and the modeling quality is monitored by reconstruction error to achieve adaptive switching of the modeling algorithm. This method effectively solves the problems of unstable accuracy and poor adaptability of traditional modeling solutions in complex dynamic scenes, and improves the quality and efficiency of three-dimensional scene reconstruction.

[0188] In another embodiment, the scene space model adaptive modeling device of multi-source data fusion can be configured separately from the central processing unit 9100. For example, the scene space model adaptive modeling device of multi-source data fusion can be configured as a chip connected to the central processing unit 9100, and the function of the scene space model adaptive modeling method of multi-source data fusion can be realized through the control of the central processing unit.

[0189] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9For components not shown, reference may be made to the prior art.

[0190] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0191] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0192] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0193] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0194] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0195] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0196] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0197] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps in the method for adaptively modeling a scene space model with multi-source data fusion in the above-mentioned embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps in the method for adaptively modeling a scene space model with multi-source data fusion in the above-mentioned embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0198] Step S101: collecting video sequences, point cloud data and IMU data of indoor and outdoor scenes, performing noise filtering and geometric correction on the data, extracting SIFT image features from the video sequence, extracting three-dimensional geometric features from the point cloud data, extracting motion posture features from the IMU data, establishing a corresponding relationship between the features, and using an adaptive weighted Kalman filter algorithm to fuse the features to obtain multi-source scene information;

[0199] Step S102: constructing a deep learning scene analysis network, performing semantic segmentation on the multi-source scene information to obtain the scene type, voxelizing the point cloud data to calculate the scene scale coefficient, using a ray projection method to count the viewpoint visibility of the multi-source scene information to obtain the occlusion degree coefficient, constructing a local curvature histogram based on the three-dimensional geometric features to calculate the geometric deformation coefficient, performing weighted summation on the scale coefficient, the occlusion degree coefficient and the geometric deformation coefficient to obtain a scene complexity score, constructing a feature vector according to the scene type and the complexity score, and retrieving the best matching modeling algorithm and parameter combination in the modeling strategy database;

[0200] Step S103: Using an optical flow algorithm to track feature points in the video sequence to obtain motion vectors, calculating the inter-frame differences of the point cloud data to generate a geometric deformation field, estimating the six-degree-of-freedom posture change of the camera based on the IMU data, inputting the motion vector, the geometric deformation field and the posture change into a recursive neural network to predict the dynamic area of ​​the scene, calculating the reconstruction error based on the modeling result of the dynamic area and the multi-source scene information, and switching the modeling algorithm to rebuild the scene model when the reconstruction error exceeds an adaptive threshold.

[0201] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application constructs complete scene information by fusing video sequences, point cloud data and IMU data. A deep learning network is used for scene analysis, and the scene complexity score is calculated based on the scale coefficient, occlusion degree and geometric deformation coefficient to achieve intelligent matching of modeling strategies. At the same time, the dynamic area of ​​the scene is predicted using the optical flow algorithm, geometric deformation field and camera pose changes, and the modeling quality is monitored by reconstruction error to achieve adaptive switching of the modeling algorithm. This method effectively solves the problems of unstable accuracy and poor adaptability of traditional modeling solutions in complex dynamic scenes, and improves the quality and efficiency of three-dimensional scene reconstruction.

[0202] The embodiments of the present application also provide a computer program product capable of implementing all the steps in the scene space model adaptive modeling method for multi-source data fusion in the above-mentioned embodiment, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the scene space model adaptive modeling method for multi-source data fusion are implemented. For example, the computer program / instruction implements the following steps:

[0203] Step S101: collecting video sequences, point cloud data and IMU data of indoor and outdoor scenes, performing noise filtering and geometric correction on the data, extracting SIFT image features from the video sequence, extracting three-dimensional geometric features from the point cloud data, extracting motion posture features from the IMU data, establishing a corresponding relationship between the features, and using an adaptive weighted Kalman filter algorithm to fuse the features to obtain multi-source scene information;

[0204] Step S102: constructing a deep learning scene analysis network, performing semantic segmentation on the multi-source scene information to obtain the scene type, voxelizing the point cloud data to calculate the scene scale coefficient, using a ray projection method to count the viewpoint visibility of the multi-source scene information to obtain the occlusion degree coefficient, constructing a local curvature histogram based on the three-dimensional geometric features to calculate the geometric deformation coefficient, performing weighted summation on the scale coefficient, the occlusion degree coefficient and the geometric deformation coefficient to obtain a scene complexity score, constructing a feature vector according to the scene type and the complexity score, and retrieving the best matching modeling algorithm and parameter combination in the modeling strategy database;

[0205] Step S103: Using an optical flow algorithm to track feature points in the video sequence to obtain motion vectors, calculating the inter-frame differences of the point cloud data to generate a geometric deformation field, estimating the six-degree-of-freedom posture change of the camera based on the IMU data, inputting the motion vector, the geometric deformation field and the posture change into a recursive neural network to predict the dynamic area of ​​the scene, calculating the reconstruction error based on the modeling result of the dynamic area and the multi-source scene information, and switching the modeling algorithm to rebuild the scene model when the reconstruction error exceeds an adaptive threshold.

[0206] From the above description, it can be seen that the computer program product provided in the embodiment of the present application constructs complete scene information by fusing video sequences, point cloud data and IMU data. A deep learning network is used for scene analysis, and the scene complexity score is calculated based on the scale coefficient, occlusion degree and geometric deformation coefficient to achieve intelligent matching of modeling strategies. At the same time, the dynamic area of ​​the scene is predicted using the optical flow algorithm, geometric deformation field and camera pose changes, and the modeling quality is monitored by reconstruction error to achieve adaptive switching of the modeling algorithm. This method effectively solves the problems of unstable accuracy and poor adaptability of traditional modeling solutions in complex dynamic scenes, and improves the quality and efficiency of three-dimensional scene reconstruction.

[0207] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0208] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0209] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0210] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0211] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A scene space model adaptive modeling method based on multi-source data fusion, characterized in that: The method comprises: Collect video sequences, point cloud data and IMU data of indoor and outdoor scenes, perform noise filtering and geometric correction on the data, extract SIFT image features from the video sequence, extract three-dimensional geometric features from the point cloud data, extract motion posture features from the IMU data, establish a corresponding relationship between the features, and use an adaptive weighted Kalman filter algorithm to fuse the features to obtain multi-source scene information; Constructing a deep learning scene analysis network, performing semantic segmentation on the multi-source scene information to obtain the scene type, voxelizing the point cloud data to calculate the scene scale coefficient, using a ray projection method to count the viewpoint visibility of the multi-source scene information to obtain the occlusion degree coefficient, constructing a local curvature histogram based on the three-dimensional geometric features to calculate the geometric deformation coefficient, performing weighted summation on the scale coefficient, the occlusion degree coefficient and the geometric deformation coefficient to obtain a scene complexity score, constructing a feature vector according to the scene type and the complexity score, and retrieving the best matching modeling algorithm and parameter combination in a modeling strategy database; The optical flow algorithm is used to track feature points in the video sequence to obtain motion vectors, the inter-frame differences of the point cloud data are calculated to generate a geometric deformation field, the six-degree-of-freedom posture change of the camera is estimated based on the IMU data, the motion vector, the geometric deformation field and the posture change are input into a recursive neural network to predict the dynamic area of ​​the scene, the reconstruction error is calculated for the modeling result of the dynamic area and the multi-source scene information, and when the reconstruction error exceeds an adaptive threshold, the modeling algorithm is switched to rebuild the scene model.

2. The scene space model adaptive modeling method of multi-source data fusion according to claim 1 is characterized in that: The method of collecting video sequences, point cloud data and IMU data of indoor and outdoor scenes, performing noise filtering and geometric correction on the data, extracting SIFT image features from the video sequence, extracting three-dimensional geometric features from the point cloud data, and extracting motion posture features from the IMU data includes: Performing image noise reduction on the collected video sequence using Gaussian filtering, filtering noise points on the point cloud data using bilateral filtering, smoothing the signal of the IMU data using wavelet transform, establishing a unified time series index for the data, and performing time stamp alignment and geometric correction on the video sequence, the point cloud data, and the IMU data; The SIFT feature points and feature descriptors are extracted from the video sequence using a scale-invariant feature transformation algorithm, the normal vector and local curvature features of the point cloud data are calculated to obtain three-dimensional geometric features, the acceleration and angular velocity of the IMU data are obtained through attitude calculation, and the motion attitude features are estimated using Kalman filtering.

3. The scene space model adaptive modeling method of multi-source data fusion according to claim 1 is characterized in that: The establishing of the correspondence between the features and fusing the features using an adaptive weighted Kalman filter algorithm to obtain multi-source scene information includes: Calculate the matching degree of feature points based on the SIFT feature descriptor, remove wrong matching points using the RANSAC algorithm, project the three-dimensional geometric features onto the image plane to establish a spatial correspondence with the SIFT feature points, and perform spatiotemporal registration of the motion posture features with the SIFT feature points and the three-dimensional geometric features; The weight coefficient of the SIFT feature is calculated according to the matching degree of the feature points, the weight coefficient is determined based on the local curvature value of the three-dimensional geometric feature, the weight coefficient is set using the estimation error of the motion posture feature, the weight coefficient is input into the Kalman filter to construct the state estimation equation, and the features are weightedly fused to obtain multi-source scene information.

4. The scene space model adaptive modeling method of multi-source data fusion according to claim 1 is characterized in that: The deep learning scene analysis network is constructed, the multi-source scene information is semantically segmented to obtain the scene type, the point cloud data is voxelized to calculate the scene scale coefficient, the viewpoint visibility of the multi-source scene information is counted by the ray projection method to obtain the occlusion degree coefficient, and the local curvature histogram is constructed based on the three-dimensional geometric features to calculate the geometric deformation coefficient, including: Constructing a U-Net convolutional neural network including an encoder and a decoder, wherein the encoder uses a residual structure to extract multi-scale features of the multi-source scene information, and the decoder restores feature resolution through upsampling and skip connections, performs pixel-level semantic segmentation on the multi-source scene information, and uses a fully connected layer to map the segmentation results to scene type labels; A three-dimensional grid is established for the point cloud data to divide voxels, the number of points in each voxel is counted to calculate the space occupancy rate to obtain the scene scale coefficient, sampling rays are projected to the scene from different viewpoints, the number of intersections between the rays and the multi-source scene information is calculated to obtain the occlusion degree coefficient, a local curvature histogram is constructed for the normal vectors and principal curvature values ​​in the three-dimensional geometric features, and the geometric deformation coefficient is calculated through the histogram peak distribution.

5. The scene space model adaptive modeling method of multi-source data fusion according to claim 1 is characterized in that: The weighted sum of the scale coefficient, the occlusion degree coefficient and the geometric deformation coefficient is performed to obtain a scene complexity score, a feature vector is constructed according to the scene type and the complexity score, and the best matching modeling algorithm and parameter combination are retrieved from a modeling strategy database, including: A normalized weight coefficient w1 is set according to the numerical range of the scale coefficient, a normalized weight coefficient w2 is determined based on the distribution characteristics of the occlusion degree coefficient, and a normalized weight coefficient w3 is determined according to the statistical characteristics of the geometric deformation coefficient, and a weighted sum operation is performed on the coefficients to obtain a scene complexity score value between 0 and 1; The scene type is converted into a type feature by one-hot encoding, the scene complexity score is converted into a numerical feature, the type feature and the numerical feature are connected to construct a high-dimensional feature vector, and the k-nearest neighbor algorithm is used to retrieve the strategy template most similar to the feature vector in a pre-built modeling strategy database to extract the corresponding modeling algorithm and parameter combination.

6. The scene space model adaptive modeling method of multi-source data fusion according to claim 1 is characterized in that: The method uses an optical flow algorithm to track feature points in the video sequence to obtain motion vectors, calculates the inter-frame differences of the point cloud data to generate a geometric deformation field, estimates the six-degree-of-freedom posture change of the camera based on the IMU data, and inputs the motion vectors, the geometric deformation field and the posture change into a recursive neural network to predict the dynamic area of ​​the scene, including: The Lucas-Kanade optical flow algorithm is used to calculate the pixel displacement field of adjacent frames of the video sequence, the corner point detection is used to screen stable feature points, the motion vector of the feature point is calculated, the Euclidean distance of the adjacent frame point set of the point cloud data is calculated to obtain the displacement vector field, the nearest neighbor search is used to establish the point correspondence relationship to generate the geometric deformation field, and the extended Kalman filter is used to perform attitude solution on the IMU data to obtain the translation vector and rotation quaternion of the camera; A long short-term memory network LSTM is constructed to model the temporal features. The feature sequence composed of the motion vector is input into the first LSTM unit, the feature sequence composed of the geometric deformation field is input into the second LSTM unit, and the feature sequence composed of the translation vector and rotation quaternion is input into the third LSTM unit. The output features of the three LSTM units are fused and a probability map of the dynamic area of ​​the scene is generated through a fully connected layer.

7. The scene space model adaptive modeling method of multi-source data fusion according to claim 1 is characterized in that: The modeling result for the dynamic area and the multi-source scene information calculate the reconstruction error, and when the reconstruction error exceeds an adaptive threshold, the modeling algorithm is switched to reconstruct the scene model, including: Binarize the probability map of the dynamic area of ​​the scene to obtain a dynamic area mask, project the modeling result onto the image plane to obtain a depth map, calculate the root mean square error and point cloud registration error between the depth map and the multi-source scene information in the dynamic area, and use an exponentially weighted average method to perform sliding statistics on the error to obtain a reconstruction error; An adaptive threshold is calculated based on historical data of the reconstruction error, and a dual threshold mechanism is used for error judgment. When the reconstruction error is greater than a high threshold or is greater than a low threshold for multiple consecutive frames, the modeling algorithm is re-retrieved in the modeling strategy database, and the retrieved new modeling algorithm is applied to scene reconstruction in dynamic areas.

8. A scene space model adaptive modeling device based on multi-source data fusion, characterized in that: The device comprises: A multi-source scene determination module is used to collect video sequences, point cloud data and IMU data of indoor and outdoor scenes, perform noise filtering and geometric correction on the data, extract SIFT image features from the video sequence, extract three-dimensional geometric features from the point cloud data, extract motion posture features from the IMU data, establish a corresponding relationship between the features, and use an adaptive weighted Kalman filter algorithm to fuse the features to obtain multi-source scene information; A scene analysis module is used to construct a deep learning scene analysis network, perform semantic segmentation on the multi-source scene information to obtain the scene type, voxelize the point cloud data to calculate the scene scale coefficient, use a ray projection method to count the viewpoint visibility of the multi-source scene information to obtain the occlusion degree coefficient, construct a local curvature histogram based on the three-dimensional geometric features to calculate the geometric deformation coefficient, perform weighted summation on the scale coefficient, the occlusion degree coefficient and the geometric deformation coefficient to obtain a scene complexity score, construct a feature vector according to the scene type and the complexity score, and retrieve the best matching modeling algorithm and parameter combination in the modeling strategy database; The adaptive modeling module is used to track the feature points in the video sequence using an optical flow algorithm to obtain motion vectors, calculate the inter-frame differences of the point cloud data to generate a geometric deformation field, estimate the six-degree-of-freedom posture change of the camera based on the IMU data, input the motion vector, the geometric deformation field and the posture change into a recursive neural network to predict the dynamic area of ​​the scene, calculate the reconstruction error based on the modeling result of the dynamic area and the multi-source scene information, and switch the modeling algorithm to reconstruct the scene model when the reconstruction error exceeds an adaptive threshold.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the scene space model adaptive modeling method of multi-source data fusion according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for adaptively modeling a scene space model by fusion of multi-source data as described in any one of claims 1 to 7 are implemented.

Images