IMU (Inertial Measurement Unit) and RGB-D (Red Green Blue-Depth) fusion driven real-time three-

Abstract

The invention discloses an IMU (Inertial Measurement Unit) and RGB-D (Red, Green and Blue-Depth) fusion-driven real-time three-dimensional modeling method and system for buildings and structures, belongs to the field of three-dimensional reconstruction, and solves the problem that a fixed sampling rate is adopted in the prior art. The method comprises the following steps: acquiring data of an RGB-D camera in real time based on an outdoor scene, wherein the data comprises IMU data acquired by an IMU sensor, a depth image acquired by the camera, a color image and a camera pose; after the sampling rate of the RGB-D camera is dynamically adjusted by a self-adaptive sampling rate adjustment algorithm based on IMU data, the RGB-D camera scans the building structure to obtain data of the RGB-D camera; recognizing and rejecting the dynamic feature points based on the result and a dynamic feature perception pose resolving optimization algorithm to obtain a static feature pose of the RGB-D camera; progressive point cloud replacement is carried out on point clouds corresponding to the depth image and the color image based on static feature poses and point cloud comprehensive precision evaluation of the RGB-D camera, and outdoor scene progressive three-dimensional modeling of real textures is carried out in combination with structural feature constraints. The method is used for real-time three-dimensional modeling of buildings and structures.

Description

Technical Field

[0001] A real-time three-dimensional modeling method and system for buildings driven by IMU and RGB-D fusion are used for real-time three-dimensional modeling of buildings and belong to the field of three-dimensional reconstruction. Background Art

[0002] With the development of technology and the expansion of applications, real-time 3D reconstruction has important value in the fields of buildings and structures. In urban planning and management, real-time 3D reconstruction plays a key role. By updating the 3D models of urban buildings, real-time monitoring of structural status can be achieved, potential problems can be discovered and solved in a timely manner, and management efficiency can be improved. In bridge inspection and maintenance, it is also very important to quickly reconstruct the 3D environment of complex structures. It can not only provide engineers with intuitive structural status, guide maintenance work, improve efficiency and safety, but also help formulate the best maintenance plan to avoid risks and waste of resources. To achieve high-precision real-time 3D reconstruction, SLAM technology is essential. SLAM technology obtains environmental data and motion information in real time through sensors (such as lidar, camera, inertial measurement unit IMU, etc.). These data are not only an important source in the real-time 3D reconstruction process, but also the basis for ensuring reconstruction accuracy and efficiency. High-precision SLAM results can significantly improve the accuracy and efficiency of 3D reconstruction, making the reconstructed 3D model more accurate and reliable.

[0003] Real-time 3D reconstruction is widely used in buildings and structures. However, due to the diversity and complexity of buildings and structures, the modeling process faces problems such as low data quality and transmission efficiency. In order to improve data quality and data transmission efficiency, the sampling rate needs to be adjusted in real time. However, most systems in existing 3D reconstruction methods (such as ORB-SLAM, LSD-SLAM, etc.) use fixed sampling rates and cannot dynamically adjust data acquisition according to scene complexity and camera motion. In fast motion, insufficient information is collected, resulting in reduced reconstruction accuracy; while in static motion, a large amount of redundant data is generated, wasting computing resources and storage space. In building scenes, the presence of dynamic objects (such as pedestrians, vehicles, etc.) will interfere with the feature extraction of the SLAM algorithm, thereby affecting the overall robustness and accuracy of the system. Existing SLAM algorithms are mainly oriented to static environments, and it is difficult to effectively identify and eliminate dynamic features, resulting in errors in feature point matching, which in turn affects positioning accuracy and environmental reconstruction effects. In addition, existing fusion algorithms (such as NDT, ICP, GICP, etc.) are inefficient when processing large-scale scenes, have insufficient ability to retain details of complex structures, and lack effective accuracy improvement mechanisms and fusion strategies when faced with low-precision point cloud data, making it difficult to achieve high-precision 3D reconstruction.

[0004] In summary, the prior art has the following technical problems:

[0005] 1. With a fixed sampling rate, data collection cannot be adjusted dynamically according to scene complexity and camera motion, resulting in insufficient information collected during fast motion, resulting in reduced reconstruction accuracy. When stationary, a large amount of redundant data is generated, wasting computing resources and storage space.

[0006] 2. Common open source SLAM algorithms (such as ORB-SLAM, LSD-SLAM, etc.) are mainly used for three-dimensional modeling in static environments. They perform well in many static environments and can provide high positioning accuracy. However, they are difficult to identify and eliminate dynamic features in dynamic environments, resulting in feature point matching errors, which in turn affects positioning accuracy and environmental reconstruction effects.

[0007] 3. Existing fusion algorithms (such as NDT, ICP, GICP, etc.) are inefficient when processing large-scale scenes, have insufficient ability to retain details of complex structures, and lack effective accuracy improvement mechanisms and fusion strategies when faced with low-precision point cloud data, making it difficult to achieve high-precision 3D reconstruction. Summary of the invention

[0008] The purpose of the present invention is to provide a real-time three-dimensional modeling method and system for buildings driven by the fusion of IMU and RGB-D, so as to solve the problem that the prior art adopts a fixed sampling rate and cannot dynamically adjust data collection according to the complexity of the scene and the camera motion, resulting in insufficient collected information during rapid movement, resulting in reduced reconstruction accuracy, and a large amount of redundant data is generated when stationary, wasting computing resources and storage space.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] A real-time three-dimensional modeling method for buildings driven by IMU and RGB-D fusion includes the following steps:

[0011] Step 1: Collect RGB-D camera data in real time based on outdoor scenes, including IMU data collected by the IMU sensor, depth images, color images and camera pose collected by the camera;

[0012] Step 2: dynamically adjust the sampling rate of the RGB-D camera based on the adaptive sampling rate adjustment algorithm of the IMU data, and based on the adjusted sampling rate, the RGB-D camera scans the building to obtain the data of the RGB-D camera;

[0013] Step 3: Based on the result obtained in step 2 and the dynamic feature perception pose solution optimization algorithm, the dynamic feature points are identified and eliminated to obtain the static feature pose of the RGB-D camera when each frame of depth image and color image is collected after optimization;

[0014] Step 4: Based on the static feature pose and comprehensive accuracy evaluation of the point cloud of the RGB-D camera, the point clouds corresponding to the depth image and the color image are progressively replaced, and the progressive 3D modeling of the outdoor scene with realistic texture is performed in combination with the structural feature constraints.

[0015] Further, the specific steps of step 2 are:

[0016] Step 2.1, based on the IMU data collected in real time by the IMU sensor within a given time, the maximum deviation ratio among the deviation ratios is selected and compared with the corresponding threshold value, wherein the data collected by the IMU sensor includes acceleration and angular velocity, and the corresponding threshold value includes a stationary threshold value and a moving threshold value;

[0017] The deviation ratio of the IMU data collected by the IMU sensor in real time within a given time includes the acceleration deviation ratio and the angular velocity deviation ratio;

[0018] The acceleration deviation ratio formula is:

[0019]

[0020] The angular velocity deviation ratio formula is:

[0021]

[0022] In the formula, A t and W t They represent the acceleration deviation ratio and angular velocity deviation ratio at time t, respectively. a(t) and ω(t) are the acceleration and angular velocity at time t collected in real time. t and ω t For the acceleration threshold and the angular velocity threshold, the maximum deviation ratio is obtained from the acceleration deviation ratio and the angular velocity deviation ratio within a given time;

[0023] Step 2.2, based on the result of comparing the maximum deviation ratio with the corresponding threshold, the motion state of the RGB-D camera when scanning the building is estimated in real time, where the motion state includes stillness, gentle movement or violent movement;

[0024] The steps to estimate the motion state of the RGB-D camera when scanning a building in real time are:

[0025] If the maximum deviation ratio is less than or equal to the static threshold, the motion state of the RGB-D camera when scanning the building is static, and the sampling rate needs to be reduced. When the RGB-D camera is always in a static state, the lower limit of the sampling rate reduction is f0 is the basic sampling rate;

[0026] If the maximum deviation ratio is less than or equal to the movement threshold, the movement state of the RGB-D camera during the scanning of the building structure is smooth movement, and the sampling rate needs to be maintained at the basic sampling rate f0;

[0027] If the maximum deviation ratio is greater than the movement threshold, the movement state of the RGB-D camera during the scanning of the building structure is violent movement, and the sampling rate needs to be increased. When the RGB-D camera is always in a violent movement state, the upper limit of the increased sampling rate is f max = 2f0;

[0028] Step 2.3: Dynamically adjust the sampling rate of the RGB-D camera based on the movement state and the adaptive sampling rate adjustment algorithm based on IMU data. Based on the adjusted sampling rate, the RGB-D camera starts to scan the building structure to obtain the data of the RGB-D camera;

[0029] The formula of the adaptive sampling rate adjustment algorithm based on IMU data is:

[0030]

[0031]

[0032] In the formula, f is the real-time sampling rate, that is, the adjusted sampling rate. When the movement state is the static state, f min ≤ f < f0. When the movement state is violent movement, f0 < f ≤ f max , k a , k ω are the adjustment coefficients of acceleration and angular velocity, σ a represents the standard deviation of acceleration within a given time, and σ ω represents the standard deviation of angular velocity within a given time.

[0033] Furthermore, the specific steps of Step 3 are as follows:

[0034] Step 3.1: Based on the data of the RGB-D camera obtained after the sampling rate adjustment, use the ORB algorithm to extract the ORB feature points of the Nth frame of depth image or color image. For the extracted ORB feature points, use the density-based spatial clustering algorithm DBSCAN to generate the feature point cluster C i , where N ≥ 1;

[0035] Step 3.2: According to each feature point cluster C i in the Nth frame of depth image or color image, obtain the centroid coordinate G i of the corresponding feature point cluster C i based on the average value of the coordinates of all feature points in it, and based on the centroid coordinate G i of this frame of feature point cluster C iThe IMU data collected in real time and adjusted by the sampling rate is used to predict the centroid coordinates of the feature point cluster with the same name in the N+1th frame depth image or color image. The prediction formula is as follows:

[0036]

[0037] In the formula, is the predicted centroid coordinate of the feature point cluster with the same name in the N+1th frame depth image or color image, R is the first integration result of the gyroscope data recorded by the IMU sensor within the frame interval, (Δx, Δy, Δz) is the second integration result of the linear acceleration recorded by the IMU sensor within the frame interval, G i is the centroid coordinate of the i-th feature point cluster;

[0038] Step 3.3, use the ORB algorithm to extract the ORB feature points of the Nth frame depth image or color image, and use the fast neighbor algorithm to match the ORB feature points extracted from the N+1th frame depth image or color image with the ORB feature points extracted from the Nth frame. The successfully matched ORB feature points are marked as pairs of points with the same name. If the ORB feature points of the Nth frame in the pair of points with the same name belong to the feature point cluster of the Nth frame, Then the feature point of the N+1th frame in the same-name point pair belongs to the feature point cluster of the N+1th frame That is, the feature point cluster with the same name in the N+1th frame relative to the Nth frame. Finally, by calculating the feature point cluster with the same name The average value of all point coordinates The centroid coordinates of the measured feature point cluster with the same name

[0039] Step 3.4, calculate the centroid coordinate deviation value ε between the measured feature point cluster with the same name and the predicted feature point cluster with the same name in the N+1th frame depth image or color image, the formula is:

[0040]

[0041] Step 3.5, if the centroid coordinate deviation value ε is greater than the set threshold, the cluster is considered to be a dynamic feature point cluster, and all ORB feature points in the cluster are removed; otherwise, the cluster is considered to be a static feature point cluster, and all ORB feature points in the cluster are retained;

[0042] Step 3.6, based on the ORB feature points in the static feature point cluster of the N+1 frame obtained in step 3.5 and the ORB feature points in the feature point cluster with the same name corresponding to the Nth frame, form a static feature point pair. According to the static feature point pair, the initial static feature pose of the RGB-D camera when collecting depth images and color images is estimated by the PnP algorithm, and then the camera pose after the sampling rate adjustment is further optimized using nonlinearity to update the camera pose of the RGB-D camera after the sampling rate adjustment, until the above steps are completed for each frame, that is, the static feature pose of the RGB-D camera when collecting each depth image and color image is obtained based on the camera pose when taking the first frame of depth image and color image.

[0043] Further, the specific steps of step 4 are:

[0044] Step 4.1: Progressive point cloud replacement

[0045] Based on the static feature pose, each pixel in the depth image or / and the color image is converted into a preliminary three-dimensional point cloud through the camera intrinsic rotation matrix, and the comprehensive accuracy of the point cloud of each frame of the preliminary three-dimensional point cloud is evaluated to perform accuracy partitioning, that is, for the Nth frame, by evaluating the comprehensive accuracy of the point cloud of each preliminary three-dimensional point cloud in the Nth frame, the point whose comprehensive accuracy of the point cloud is lower than the threshold is used as a replacement target, and the replacement target is marked after it is found, and the corresponding preliminary three-dimensional point cloud in the N+1 frame is found through the feature matching technology, so that the point clouds of two adjacent frames are accurately aligned, and the comprehensive accuracy of the point cloud of the preliminary three-dimensional point cloud corresponding to the N+1 frame is evaluated, and the comprehensive accuracy of the point cloud of the replacement target and its corresponding point in the adjacent frames is compared between frames. If the comprehensive accuracy of the point cloud of the N+1th frame is lower than that of the Nth frame, the point in the Nth frame is deleted, and if the comprehensive accuracy of the point cloud of the N+1th frame is higher than that of the Nth frame, the corresponding preliminary three-dimensional point cloud in the N+1th frame is used as the replacement point, and the attribute data of the replacement point in the N+1 frame is used as the attribute information of the replacement target in the Nth frame to complete the replacement, wherein the feature matching technology includes the ICP algorithm;

[0046] The point cloud comprehensive accuracy evaluation formula of each preliminary 3D point cloud is:

[0047]

[0048] Among them, S m represents the comprehensive accuracy evaluation value of each preliminary 3D point cloud converted from the m-th frame depth image or / and color image, E m is the reprojection error formula, that is, E m =‖P(x m ,y m ,z m )-(u m ,v m )‖, where P is the projection matrix, (x m ,ym ,z m ) Preliminary 3D point cloud coordinates, (u m ,v m ) Two-dimensional point cloud projection coordinates, N m is the average neighbor distance method, Calculate the average distance between point m and its neighboring point j as the point density accuracy, D m is the depth accuracy of the mth point, d m is the depth value of the mth point, is the depth change rate of the mth point, let is the depth value d between the mth point and its neighboring point j j The average difference of , w1, w2 and w3 are weight parameters used to measure the importance of depth accuracy, reprojection error and point density in the comprehensive score, k represents the total number of preliminary 3D point clouds in the Nth frame, K represents the total variable of the preliminary 3D point cloud in the Nth frame;

[0049] Step 4.2: Generate a mesh model based on the result obtained in step 4.1, the point cloud fusion optimization method with structural feature constraints and the Poisson estimation method, that is, perform progressive 3D modeling of outdoor scenes with real textures. The specific steps are as follows:

[0050] According to the characteristics of buildings and structures, the basic data of multi-source buildings and structures are pre-integrated and stored based on prior knowledge and expert guidance to obtain a feature library. The structural features in the feature library include plumbness, edge-corner point, topology-orientation and continuity;

[0051] Taking the point cloud corresponding to the depth image and color image after substitution optimization as the data basis, the structured data is parsed under the constraint of the feature library, that is, the principal component analysis method is used to identify and extract the structured point cloud data in the data basis that can match the features in the feature library. After extraction, the principal component analysis method is used to extract the same geometric structure in two adjacent frames, and then the feature matching method is used to match the same geometric structure features between two adjacent frames, so as to obtain the feature point pairs between frames;

[0052] The rotation and translation matrices are calculated based on the feature point pairs between frames, and the point clouds of adjacent frames are aligned based on the rotation and translation matrices. After alignment, the ICP algorithm is used to iteratively optimize the correspondence between the two point clouds. If the convergence conditions are met, the final fine alignment result is obtained. Otherwise, the rotation and translation matrices are calculated again to align the point clouds of adjacent frames.

[0053] Based on the feature point pairs between frames obtained after fine alignment, point cloud fusion is performed by combining the voxel grid method and the nearest neighbor search to optimize the fusion of point clouds in the overlapping areas of adjacent frame spaces. At the same time, in the boundary area, based on the feature point pairs between frames obtained after fine alignment, the boundary recognition and surface fitting method or the Laplace smoothing method are used to optimize the point cloud boundary fusion in the smooth boundary area. After optimization, the Poisson estimation method is finally used to perform progressive 3D modeling of outdoor scenes with real textures.

[0054] A real-time 3D modeling system for buildings driven by IMU and RGB-D fusion, comprising:

[0055] Collection end:

[0056] Collect RGB-D camera data in real time based on outdoor scenes, including IMU data collected by IMU sensors, depth images, color images and camera poses collected by cameras;

[0057] The sampling rate of the RGB-D camera is dynamically adjusted by an adaptive sampling rate adjustment algorithm based on IMU data, and based on the adjusted sampling rate, the RGB-D camera scans the building to obtain the data of the RGB-D camera;

[0058] Server side:

[0059] Based on the RGB-D camera scanning the building after adjusting the sampling rate to obtain the RGB-D camera data and the dynamic feature perception pose solution optimization algorithm to identify and eliminate dynamic feature points, the static feature pose of the RGB-D camera when collecting each frame of depth image and color image after optimization is obtained;

[0060] Based on the static feature pose and point cloud comprehensive accuracy evaluation of the RGB-D camera, the point clouds corresponding to the depth image and color image are progressively replaced, and the progressive 3D modeling of outdoor scenes with realistic textures is performed in combination with structural feature constraints.

[0061] Client: Visualize and view the progressive 3D modeling of outdoor scenes built on the server.

[0062] Furthermore, the sampling rate of the RGB-D camera is dynamically adjusted by the adaptive sampling rate adjustment algorithm based on the IMU data in the acquisition end, and based on the adjusted sampling rate, the RGB-D camera scans the building to obtain the data of the RGB-D camera in the specific steps as follows:

[0063] Step 2.1, based on the IMU data collected in real time by the IMU sensor within a given time, the maximum deviation ratio among the deviation ratios is selected and compared with the corresponding threshold value, wherein the data collected by the IMU sensor includes acceleration and angular velocity, and the corresponding threshold value includes a stationary threshold value and a moving threshold value;

[0064] The deviation ratio of the IMU data collected by the IMU sensor in real time within a given time includes the acceleration deviation ratio and the angular velocity deviation ratio;

[0065] The acceleration deviation ratio formula is:

[0066]

[0067] The angular velocity deviation ratio formula is:

[0068]

[0069] In the formula, A t and W t They represent the acceleration deviation ratio and angular velocity deviation ratio at time t, respectively. a(t) and ω(t) are the acceleration and angular velocity at time t collected in real time. t and ω t For the acceleration threshold and the angular velocity threshold, the maximum deviation ratio is obtained from the acceleration deviation ratio and the angular velocity deviation ratio within a given time;

[0070] Step 2.2: Based on the result of comparing the maximum deviation ratio with the corresponding threshold, the motion state of the RGB-D camera when scanning the building is estimated in real time, where the motion state includes stillness, gentle movement or violent movement. The steps of estimating the motion state of the RGB-D camera when scanning the building in real time are:

[0071] If the maximum deviation ratio is less than or equal to the static threshold, the motion state of the RGB-D camera when scanning the building is static, and the sampling rate needs to be reduced. When the RGB-D camera is always in a static state, the lower limit of the sampling rate reduction is f0 is the basic sampling rate;

[0072] If the maximum deviation ratio is less than or equal to the motion threshold, the motion state of the RGB-D camera when scanning the building is smooth movement, and the sampling rate needs to be maintained at the basic sampling rate f0;

[0073] If the maximum deviation ratio is greater than the motion threshold, the motion state of the RGB-D camera when scanning the building is violent motion, and the sampling rate needs to be increased. When the RGB-D camera is in a violent motion state, the upper limit of the sampling rate is f max =2f0;

[0074] Step 2.3: The sampling rate of the RGB-D camera is dynamically adjusted based on the motion state and the adaptive sampling rate adjustment algorithm based on the IMU data. Based on the adjusted sampling rate, the RGB-D camera starts scanning the building to obtain the data of the RGB-D camera. The formula of the adaptive sampling rate adjustment algorithm based on the IMU data is:

[0075]

[0076] In the formula, f is the real-time sampling rate, that is, the adjusted sampling rate. When the motion state is the static state, f min ≤f < f0. When the motion state is intense exercise, f0 < f ≤ f max , k a , k ω are the acceleration and angular velocity adjustment coefficients, and σ a represents the standard deviation of the acceleration within a given time, and σ ω represents the standard deviation of the angular velocity within a given time.

[0077] Furthermore, the specific steps for the server side to identify and remove dynamic feature points based on the data of the RGB-D camera obtained after adjusting the sampling rate and the dynamic feature perception pose solution optimization algorithm, and obtain the static feature pose of the RGB-D camera when collecting each frame of depth image and color image after optimization are as follows:

[0078] Step 3.1: Based on the data of the RGB-D camera obtained after sampling rate adjustment, use the ORB algorithm to extract the ORB feature points of the Nth frame of depth image or color image, and use the density-based spatial clustering algorithm DBSCAN to generate a feature point cluster C for the extracted ORB feature points i , where N ≥ 1;

[0079] Step 3.2: According to each feature point cluster C in the Nth frame of depth image or color image i , obtain the centroid coordinate G of the corresponding feature point cluster C based on the average value of the coordinates of all feature points within it i , and based on the centroid coordinate G of this frame of feature point cluster C i and the IMU data after sampling rate adjustment collected in real time, predict the centroid coordinate of the corresponding feature point cluster of the (N + 1)th frame of depth image or color image i The prediction formula is as follows: i

[0080]

[0081] In the formula,

[0082] is the predicted centroid coordinate of the corresponding feature point cluster of the (N + 1)th frame of depth image or color image, R is the result of a single integration of the gyroscope data recorded by the IMU sensor within the frame interval time, (Δx, Δy, Δz) is the result of the second integration of the linear acceleration recorded by the IMU sensor within the frame interval time, and G i is the centroid coordinate of the i-th feature point cluster;

[0082] Step 3.3, use the ORB algorithm to extract the ORB feature points of the Nth frame depth image or color image, and use the fast neighbor algorithm to match the ORB feature points extracted from the N+1th frame depth image or color image with the ORB feature points extracted from the Nth frame. The successfully matched ORB feature points are marked as pairs of points with the same name. If the ORB feature points of the Nth frame in the pair of points with the same name belong to the feature point cluster of the Nth frame, Then the feature point of the N+1th frame in the same-name point pair belongs to the feature point cluster of the N+1th frame That is, the feature point cluster with the same name in the N+1th frame relative to the Nth frame. Finally, by calculating the feature point cluster with the same name The average value of all point coordinates The centroid coordinates of the measured feature point cluster with the same name

[0083] Step 3.4, calculate the centroid coordinate deviation value ε between the measured feature point cluster with the same name and the predicted feature point cluster with the same name in the N+1th frame depth image or color image, the formula is:

[0084]

[0085] Step 3.5, if the centroid coordinate deviation value ε is greater than the set threshold, the cluster is considered to be a dynamic feature point cluster, and all ORB feature points in the cluster are removed; otherwise, the cluster is considered to be a static feature point cluster, and all ORB feature points in the cluster are retained;

[0086] Step 3.6, based on the ORB feature points in the static feature point cluster of the N+1 frame obtained in step 3.5 and the ORB feature points in the feature point cluster with the same name corresponding to the Nth frame, form a static feature point pair. According to the static feature point pair, the initial static feature pose of the RGB-D camera when collecting depth images and color images is estimated by the PnP algorithm, and then the camera pose after the sampling rate adjustment is further optimized using nonlinearity to update the camera pose of the RGB-D camera after the sampling rate adjustment, until the above steps are completed for each frame, that is, the static feature pose of the RGB-D camera when collecting each depth image and color image is obtained based on the camera pose when taking the first frame of depth image and color image.

[0087] Furthermore, the specific steps of progressively replacing the point clouds corresponding to the depth image and the color image based on the static feature pose of the RGB-D camera and the comprehensive accuracy evaluation of the point cloud in the server side, and performing progressive 3D modeling of the outdoor scene with real texture in combination with the structural feature constraints are as follows:

[0088] Step 4.1: Progressive point cloud replacement

[0089] Based on the static feature pose, each pixel in the depth image or / and the color image is converted into a preliminary three-dimensional point cloud through the camera intrinsic rotation matrix, and the comprehensive accuracy of the point cloud of each frame of the preliminary three-dimensional point cloud is evaluated to perform accuracy partitioning, that is, for the Nth frame, by evaluating the comprehensive accuracy of the point cloud of each preliminary three-dimensional point cloud in the Nth frame, the point whose comprehensive accuracy of the point cloud is lower than the threshold is used as a replacement target, and the replacement target is marked after it is found, and the corresponding preliminary three-dimensional point cloud in the N+1 frame is found through the feature matching technology, so that the point clouds of two adjacent frames are accurately aligned, and the comprehensive accuracy of the point cloud of the preliminary three-dimensional point cloud corresponding to the N+1 frame is evaluated, and the comprehensive accuracy of the point cloud of the replacement target and its corresponding point in the adjacent frames is compared between frames. If the comprehensive accuracy of the point cloud of the N+1th frame is lower than that of the Nth frame, the point in the Nth frame is deleted, and if the comprehensive accuracy of the point cloud of the N+1th frame is higher than that of the Nth frame, the corresponding preliminary three-dimensional point cloud in the N+1th frame is used as the replacement point, and the attribute data of the replacement point in the N+1 frame is used as the attribute information of the replacement target in the Nth frame to complete the replacement, wherein the feature matching technology includes the ICP algorithm;

[0090] The point cloud comprehensive accuracy evaluation formula of each preliminary 3D point cloud is:

[0091]

[0092] Among them, S m represents the comprehensive accuracy evaluation value of each preliminary 3D point cloud converted from the m-th frame depth image or / and color image, E m is the reprojection error formula, that is, E m =‖P(x m ,y m ,z m )-(u m ,v m )‖, where P is the projection matrix, (x m ,y m ,z m ) Preliminary 3D point cloud coordinates, (u m ,v m ) Two-dimensional point cloud projection coordinates, N m is the average neighbor distance method, Calculate the average distance between point m and its neighboring point j as the point density accuracy, D m is the depth accuracy of the mth point, d m is the depth value of the mth point, is the depth change rate of the mth point, let is the depth value d between the mth point and its neighboring point j jThe average difference of , w1, w2 and w3 are weight parameters used to measure the importance of depth accuracy, reprojection error and point density in the comprehensive score, k represents the total number of preliminary 3D point clouds in the Nth frame, K represents the total variable of the preliminary 3D point cloud in the Nth frame;

[0093] Step 4.2: Generate a mesh model based on the result obtained in step 4.1, the point cloud fusion optimization method with structural feature constraints and the Poisson estimation method, that is, perform progressive 3D modeling of outdoor scenes with real textures. The specific steps are as follows:

[0094] According to the characteristics of buildings and structures, the basic data of multi-source buildings and structures are pre-integrated and stored based on prior knowledge and expert guidance to obtain a feature library. The structural features in the feature library include plumbness, edge-corner point, topology-orientation and continuity;

[0095] Taking the point cloud corresponding to the depth image and color image after substitution optimization as the data basis, the structured data is parsed under the constraint of the feature library, that is, the principal component analysis method is used to identify and extract the structured point cloud data in the data basis that can match the features in the feature library. After extraction, the principal component analysis method is used to extract the same geometric structure in two adjacent frames, and then the feature matching method is used to match the same geometric structure features between two adjacent frames, so as to obtain the feature point pairs between frames;

[0096] The rotation and translation matrices are calculated based on the feature point pairs between frames, and the point clouds of adjacent frames are aligned based on the rotation and translation matrices. After alignment, the ICP algorithm is used to iteratively optimize the correspondence between the two point clouds. If the convergence conditions are met, the final fine alignment result is obtained. Otherwise, the rotation and translation matrices are calculated again to align the point clouds of adjacent frames.

[0097] Based on the feature point pairs between frames obtained after fine alignment, point cloud fusion is performed by combining the voxel grid method and the nearest neighbor search to optimize the fusion of point clouds in the overlapping areas of adjacent frame spaces. At the same time, in the boundary area, based on the feature point pairs between frames obtained after fine alignment, the boundary recognition and surface fitting method or the Laplace smoothing method are used to optimize the point cloud boundary fusion in the smooth boundary area. After optimization, the Poisson estimation method is finally used to perform progressive 3D modeling of outdoor scenes with real textures.

[0098] Compared with the prior art, the advantages of the present invention are:

[0099] 1. The present invention dynamically adjusts the sampling rate by using the data provided by the IMU sensor, thereby overcoming the problems of data redundancy and low data transmission efficiency caused by the use of a fixed sampling rate in the existing 3D reconstruction method, that is, overcoming the problem that when the camera moves quickly, insufficient information is collected, resulting in reduced reconstruction accuracy, and when the camera is stationary, a large amount of redundant data is generated, wasting computing resources and storage space;

[0100] Second, the IMU sensor-assisted dynamic feature perception posture solution optimization algorithm in the present invention uses the data provided by the IMU to perceive and eliminate dynamic features, thereby solving the problems of feature point matching errors and low posture solution accuracy caused by the inability of existing SLAM algorithms to effectively identify and eliminate dynamic features in dynamic environments, thereby effectively improving positioning accuracy and environmental reconstruction effects;

[0101] 3. The progressive 3D modeling method of underground space scenes constrained by structural features in the present invention makes up for the problems of low consistency and accuracy of 3D models caused by the lack of low-precision point cloud optimization strategy in existing fusion algorithms by replacing low-precision point cloud data and fusing them with refined boundaries. That is, it overcomes the problems of low efficiency in processing large-scale scenes and insufficient ability to retain details of complex structures. It has an effective accuracy improvement mechanism and fusion strategy when facing low-precision point cloud data, thereby achieving high-precision 3D reconstruction.

[0102] Fourth, the data of the RGB-D camera in the present invention is combined with the IMU sensor and camera data, the IMU data is used to dynamically adjust the sampling rate of the RGB-D camera, the IMU data and image feature points are combined to identify and remove dynamic feature points, and finally the image modeling collected by the RGB-D camera can improve the system's adaptability in different environments, lay the foundation for the future integration of other types of sensors, and open up new possibilities for 3D reconstruction in more complex and diverse environments in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0104] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0105] Figure 2 This is a flow chart of dynamic sampling rate adjustment in the present invention;

[0106] Figure 3 A schematic diagram of an adaptive sampling rate adjustment algorithm of the present invention;

[0107] Figure 4 It is a schematic diagram of a dynamic feature perception method for feature point cluster center prediction and deviation detection in the present invention;

[0108] Figure 5 It is a schematic diagram of progressive three-dimensional modeling of spatial structure constraints in the present invention;

[0109] Figure 6 It is a diagram of the gradual replacement process of the present invention;

[0110] Figure 7 It is the point cloud fusion optimization process constrained by structural features in the present invention;

[0111] Figure 8 It is a real-time three-dimensional modeling system of buildings and structures driven by the fusion of IMU and RGB-D in the present invention. DETAILED DESCRIPTION

[0112] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0113] A real-time three-dimensional modeling method for buildings driven by IMU and RGB-D fusion includes the following steps:

[0114] Collect RGB-D camera data in real time based on outdoor scenes, including IMU data collected by IMU sensors, depth images, color images and camera poses collected by cameras;

[0115] An adaptive sampling rate adjustment algorithm based on IMU data dynamically adjusts the sampling rate of the RGB-D camera, and based on the adjusted sampling rate, the RGB-D camera scans the building to obtain the data of the RGB-D camera.

[0116] The present invention uses the acceleration and angular velocity data collected by the IMU sensor and the depth image and color image collected by the RGB-D camera to estimate the motion state of the camera when scanning the building in real time by comparing the maximum deviation ratio with the corresponding threshold, including stillness, slow movement, and violent movement. Based on this motion state information, the sampling rate of the RGB-D camera is dynamically adjusted. The sampling rate adjustment process is as follows: Figure 2As shown in the figure, when the camera is stationary, such as when scanning the details of a building, the sampling rate can be lowered to reduce data redundancy and power consumption. When the camera moves slowly and steadily, such as when scanning the surface of a building, an appropriate sampling rate is maintained. When the camera moves violently, the sampling rate is greatly increased to obtain more timing information and reduce data loss due to motion blur. Through this adaptive sampling strategy, the sampling parameters are dynamically adjusted according to the real-time motion state. Under the premise of ensuring the quality of 3D reconstruction, unnecessary data collection is minimized, thereby improving the overall efficiency and real-time performance of the system. Finally, the collected depth image (depth), RGB image (i.e., color image) and camera posture (posture) are output to provide reliable data support for subsequent applications. As shown in the figure, the sampling rate is greatly increased to obtain more timing information and reduce data loss due to motion blur. Figure 3 As shown, the specific steps are:

[0117] Based on the IMU data collected in real time by the IMU sensor within a given time, the maximum deviation ratio among the deviation ratios is selected and compared with the corresponding threshold value, wherein the data collected by the IMU sensor includes acceleration and angular velocity, and the corresponding threshold value includes a stationary threshold value and a moving threshold value;

[0118] The deviation ratio of the IMU data collected by the IMU sensor in real time within a given time includes the acceleration deviation ratio and the angular velocity deviation ratio;

[0119] The acceleration deviation ratio formula is:

[0120]

[0121] The angular velocity deviation ratio formula is:

[0122]

[0123] In the formula, A t and W t They represent the acceleration deviation ratio and angular velocity deviation ratio at time t, respectively. a(t) and ω(t) are the acceleration and angular velocity at time t collected in real time. t and ω t For the acceleration threshold and the angular velocity threshold, the maximum deviation ratio is obtained from the acceleration deviation ratio and the angular velocity deviation ratio within a given time;

[0124] Based on the result of comparing the maximum deviation ratio with the corresponding threshold, the motion state of the RGB-D camera when scanning the building is estimated in real time, where the motion state includes stillness, gentle movement or violent movement. The steps of estimating the motion state of the RGB-D camera when scanning the building in real time are:

[0125] If the maximum deviation ratio is less than or equal to the static threshold, the motion state of the RGB-D camera during the scanning of the building structure is the static state, and the sampling rate needs to be reduced. When the RGB-D camera is always in the static state, the lower limit of the sampling rate reduction is f0 is the basic sampling rate;

[0126] If the maximum deviation ratio is less than or equal to the moving threshold, the motion state of the RGB-D camera during the scanning of the building structure is gentle movement, and the sampling rate needs to be maintained at the basic sampling rate f0;

[0127] If the maximum deviation ratio is greater than the moving threshold, the motion state of the RGB-D camera during the scanning of the building structure is violent movement, and the sampling rate needs to be increased. When the RGB-D camera is always in the violent movement state, the upper limit of the sampling rate increase is f max = 2f0;

[0128] Dynamically adjust the sampling rate of the RGB-D camera based on the motion state and the adaptive sampling rate adjustment algorithm based on IMU data. Based on the adjusted sampling rate, the RGB-D camera starts to scan the building structure to obtain the data of the RGB-D camera; the formula of the adaptive sampling rate adjustment algorithm based on IMU data is:

[0129]

[0130] In the formula, f is the real-time sampling rate, that is, the adjusted sampling rate. When the motion state is the static state, f min ≤ f < f0. When the motion state is violent movement, f0 < f ≤ f max , k a , k ω are the acceleration and angular velocity adjustment coefficients, and σ a represents the standard deviation of the acceleration within a given time, and σ ω represents the standard deviation of the angular velocity within a given time.

[0131] Based on the adjusted sampling rate, the RGB-D camera scans the building structure to obtain the data of the RGB-D camera and the dynamic feature perception pose solution optimization algorithm identifies and eliminates the dynamic feature points, and obtains the static feature pose of the RGB-D camera when collecting each frame of depth image and color image after optimization.

[0132] During the spatial data collection process of buildings and structures, there is a high probability that scenes containing dynamic targets such as pedestrians, flying birds or sports vehicles will appear. The feature points matched on such dynamic targets have a negative impact on the accuracy of pose solution. In order to simply and efficiently meet the accuracy requirements, this case proposes a dynamic feature perception pose solution optimization algorithm. By combining IMU data and spatial features, it realizes accurate perception and elimination of dynamic targets, thereby optimizing pose solution. This case proposes to first perform density clustering on the feature points, and effectively reduces computational redundancy by centring the feature point clusters. At the same time, it takes into account the spatial structure information, so that the dynamic feature points are filtered to a high degree. Compared with the method of eliminating feature points one by one, this method can avoid the omission problem caused by some feature points being stationary within a specific time period. By deleting the entire feature point cluster at one time, even if some of the feature points are stationary at certain times, as long as the points in its field move, these feature points will be effectively deleted, thereby significantly improving the thoroughness of filtering.

[0133] This case adopts the dynamic feature perception method of feature point cluster center prediction and deviation detection to identify and remove dynamic feature points, thereby improving the estimation accuracy of device posture parameters. The specific steps are:

[0134] Based on the data of the RGB-D camera obtained after the sampling rate is adjusted, the ORB algorithm is used to extract the ORB feature points of the Nth frame depth image or color image. The density-based spatial clustering algorithm DBSCAN is used to generate feature point clusters C for the extracted ORB feature points. i , where N ≥ 1;

[0135] According to each feature point cluster C in the Nth frame depth image or color image i The average value of all feature point coordinates in the corresponding feature point cluster C is obtained i The center of gravity coordinates G i , and based on the frame feature point cluster C i The center of gravity coordinates G i The IMU data collected in real time and adjusted by the sampling rate is used to predict the centroid coordinates of the feature point cluster with the same name in the N+1th frame depth image or color image. The prediction formula is as follows:

[0136]

[0137] In the formula, is the predicted centroid coordinate of the feature point cluster with the same name in the N+1th frame depth image or color image. R is the result of a single integration of the gyroscope data recorded by the IMU sensor within the frame interval. If the sampling rate of the IMU is 100Hz and the sampling rate of the RGB-D camera is 20fps, 5 sets of gyroscope data can be obtained within the frame interval. Each set of data contains three values, corresponding to the angular velocity around the X-axis, Y-axis, and Z-axis, respectively, in radians per second. The rotation vector is calculated for each set of gyroscope data, and then the Rodriguez formula is used to convert the rotation vector into an intermediate rotation matrix dR. The rotation matrix within the frame interval is obtained by multiplication and iteration, which is the result of a single integration. (Δx, Δy, Δz) is the result of a secondary integration of the linear acceleration recorded by the IMU sensor within the frame interval, that is, through the 5 sets of accelerometer data within the frame interval, the 5 intermediate displacements are calculated by secondary integration, and finally the inter-frame translation vector is obtained by summing, which is the result of the secondary integration, G i is the centroid coordinate of the i-th feature point cluster;

[0138] The ORB algorithm is used to extract the ORB feature points of the Nth frame depth image or color image. According to the ORB feature points extracted from the N+1th frame depth image or color image and the ORB feature points extracted from the Nth frame, the fast neighbor algorithm is used to match the ORB feature points. The successfully matched ORB feature points are marked as pairs of points with the same name. If the ORB feature points of the Nth frame in the pair of points with the same name belong to the feature point cluster of the Nth frame, Then the feature point of the N+1th frame in the same-name point pair belongs to the feature point cluster of the N+1th frame That is, the feature point cluster with the same name in the N+1th frame relative to the Nth frame. Finally, by calculating the feature point cluster with the same name The average value of all point coordinates The centroid coordinates of the measured feature point cluster with the same name

[0139] Calculate the centroid coordinate deviation value ε of the measured feature point cluster with the same name and the predicted feature point cluster with the same name in the N+1th frame depth image or color image. The formula is:

[0140]

[0141] If the centroid coordinate deviation value ε is greater than the set threshold, the cluster is considered to be a dynamic feature point cluster, and all ORB feature points in the cluster are removed; otherwise, the cluster is considered to be a static feature point cluster, and all ORB feature points in the cluster are retained;

[0142] Based on the ORB feature points in the static feature point cluster of the N+1 frame obtained and the ORB feature points in the feature point cluster of the same name corresponding to the Nth frame, a static feature point pair is formed. Based on the static feature point pair, the initial static feature pose of the RGB-D camera when collecting depth images and color images is estimated by the PnP algorithm, and then the camera pose after the sampling rate adjustment is further optimized using nonlinearity to update the camera pose of the RGB-D camera after the sampling rate adjustment, until the above steps are completed for each frame, that is, the static feature pose of the RGB-D camera when collecting each depth image and color image is obtained based on the camera pose when shooting the first frame of depth image and color image. The device pose parameters are updated only according to the static feature points to avoid the interference of dynamic feature points on pose solution.

[0143] Through the above method, static feature points can be more accurately identified and utilized, the pose estimation accuracy can be optimized, and ultimately the 3D modeling accuracy can be improved.

[0144] Based on the static feature pose and comprehensive accuracy evaluation of the point cloud of the RGB-D camera, the point clouds corresponding to the depth image and the color image are progressively replaced, and the progressive 3D modeling of outdoor scenes with realistic textures is performed in combination with structural feature constraints.

[0145] As the core of 3D reconstruction, the 3D modeling method aims to transform discrete point cloud data into a continuous model to achieve visual expression of 3D scenes. At present, the NDT fusion algorithm is sensitive to initial alignment, has low efficiency when processing large-scale scenes, and has insufficient ability to retain details of complex structures. This leads to a lack of effective precision improvement mechanisms and fusion strategies when facing low-precision point cloud data, making it difficult to ensure the accuracy and integrity of the final model. Therefore, this case proposes a progressive 3D modeling method based on structural feature constraints. By selectively replacing and finely fusing traditional point cloud data, each step is improved and perfected on the basis of the previous step, so as to achieve a gradual improvement in the accuracy and integrity of the model.

[0146] The specific process is as follows Figure 5 As shown, based on the depth image or color image, the reprojection error and the point density parameters, the point accuracy is analyzed and the accuracy threshold is set, and the points with lower accuracy are screened as replacement targets and marked. The replacement target of the Nth frame is used to match the high-precision replacement points of the next frame, while inheriting and transferring the attributes to ensure that the attributes of the high-precision points can be seamlessly integrated into the original point cloud, thereby repeatedly achieving progressive optimization of the point cloud between frames. In the fusion stage, the replacement points and the replacement targets are aligned and replaced to initially form a point cloud model. In order to further improve the accuracy of the model, this case also proposes a fine fusion step: identify the spatial structural features in the point cloud, including corners, edges, and planes, and match the spatial structural features in two adjacent frames. The matching results are used for boundary fine fusion to ensure the complete structural features of the point cloud model and improve the matching accuracy. The specific steps are:

[0147] Progressive point cloud replacement

[0148] Based on the static feature pose, each pixel in the depth image or / and the color image is converted into a preliminary three-dimensional point cloud through the camera intrinsic rotation matrix, and the comprehensive accuracy of the point cloud of each frame of the preliminary three-dimensional point cloud is evaluated to perform accuracy partitioning, that is, for the Nth frame, by evaluating the comprehensive accuracy of the point cloud of each preliminary three-dimensional point cloud in the Nth frame, the point whose comprehensive accuracy of the point cloud is lower than the threshold is used as the replacement target, and the replacement target is marked after it is found, and the corresponding preliminary three-dimensional point cloud in the N+1 frame is found through the feature matching technology, so that the point clouds of two adjacent frames are accurately aligned, and the comprehensive accuracy of the point cloud of the preliminary three-dimensional point cloud corresponding to the N+1 frame is evaluated, and the comprehensive accuracy of the point cloud of the replacement target and its corresponding point in the adjacent frames is compared between frames. If the comprehensive accuracy of the point cloud of the N+1th frame is lower than that of the Nth frame, the point in the Nth frame is deleted, and if the comprehensive accuracy of the point cloud of the N+1th frame is higher than that of the Nth frame, the corresponding preliminary three-dimensional point cloud in the N+1th frame is used as the replacement point, and the attribute data of the replacement point in the N+1 frame is used as the attribute information of the replacement target in the Nth frame to complete the replacement, wherein the feature matching technology includes the ICP algorithm; Figure 6 As shown in the figure, for the depth value, let A1 and A2 be the real points, and A1', A2' be the observed points. As the depth increases, the error becomes larger. For the reprojection error, the three-dimensional coordinates of the points are converted into two-dimensional coordinates according to the internal and external parameters of the camera, and the comparison with the real two-dimensional projection can also be used as the comprehensive accuracy evaluation of the point cloud. The point density also directly affects the coverage and accuracy of the point cloud, so the three are used as the basis for determining the threshold.

[0149] The point cloud comprehensive accuracy evaluation formula of each preliminary 3D point cloud is:

[0150]

[0151] Among them, S m represents the comprehensive accuracy evaluation value of each preliminary 3D point cloud converted from the m-th frame depth image or / and color image, E m is the reprojection error formula, that is, E m =‖P(x m ,y m ,z m )-(u m ,v m )‖, where P is the projection matrix, (x m ,y m ,z m ) Preliminary 3D point cloud coordinates, (u m ,v m ) Two-dimensional point cloud projection coordinates, N m is the average neighbor distance method, Calculate the average distance between point m and its neighboring point j as the point density accuracy, D m is the depth accuracy of the mth point, d m is the depth value of the mth point, is the depth change rate of the mth point, let is the depth value d between the mth point and its neighboring point j j The average difference of , w1, w2 and w3 are weight parameters used to measure the importance of depth accuracy, reprojection error and point density in the comprehensive score, k represents the total number of preliminary 3D point clouds in the Nth frame, K represents the total variable of the preliminary 3D point cloud in the Nth frame;

[0152] The progressive point cloud replacement method improves the accuracy and consistency of the overall point cloud data by gradually replacing and optimizing the low-precision point cloud, providing a reliable data foundation for subsequent fine 3D modeling.

[0153] According to the characteristics of buildings and structures, complex multi-source basic data are pre-integrated and stored. The knowledge reference mainly comes from the prior knowledge of residential buildings and bridges and other structures, relevant specification documents and expert guidance. Through analysis and summary, plumb constraints, edge-corner constraints, topology-orientation constraints and continuity constraints are obtained. At the same time, the point cloud that has been replaced and optimized is used as the data basis, and the structured data is parsed under the constraints of the feature library, that is, the fine extraction of spatial features is achieved under the constraints of structural features, and then the feature point pairs are obtained by matching features between frames. Finally, under the constraints of the matching results, the overall point cloud is adjusted and aligned to achieve further fine fusion, and the replacement boundary fusion effect is optimized to obtain more complete point cloud data. On this basis, mesh construction, texture mapping, detail rendering and other operations are carried out to complete the modeling and improve the accuracy of the three-dimensional model. Figure 7 shown.

[0154] Based on the point cloud of the depth image and color image after substitution optimization, the point cloud fusion optimization method with structural feature constraints and the Poisson estimation method, a mesh model is generated, that is, progressive 3D modeling of outdoor scenes with real textures is performed. The specific steps are as follows:

[0155] According to the characteristics of buildings and structures, the basic data of multi-source buildings and structures are pre-integrated and stored based on prior knowledge and expert guidance to obtain a feature library. The structural features in the feature library include plumbness, edge-corner point, topology-orientation and continuity;

[0156] Taking the point cloud corresponding to the depth image and color image after substitution optimization (the point cloud includes the information of the depth image and the color image (RGB image)) as the data basis, the structured data is parsed under the constraint of the feature library, that is, the principal component analysis method is used to identify and extract the structured point cloud data in the data basis that can match the features in the feature library. After extraction, the principal component analysis method is used to extract the same geometric structure in two adjacent frames, and then the feature matching method is used to match the same geometric structure features between the two adjacent frames, so as to obtain the feature point pairs between the frames;

[0157] The rotation and translation matrices are calculated based on the feature point pairs between frames, and the point clouds of adjacent frames are aligned based on the rotation and translation matrices. After alignment, the ICP algorithm is used to iteratively optimize the correspondence between the two point clouds. If the convergence conditions are met, the final fine alignment result is obtained. Otherwise, the rotation and translation matrices are calculated again to align the point clouds of adjacent frames.

[0158] Based on the feature point pairs between frames obtained after fine alignment, point cloud fusion is performed by combining the voxel grid method and the nearest neighbor search to optimize the fusion of point clouds in the overlapping areas of adjacent frame spaces. At the same time, in the boundary area, based on the feature point pairs between frames obtained after fine alignment, the boundary recognition and surface fitting method or the Laplace smoothing method are used to optimize the point cloud boundary fusion in the smooth boundary area. After optimization, the Poisson estimation method is finally used to perform progressive 3D modeling of outdoor scenes with real textures.

[0159] Realization of real-time 3D modeling system for buildings driven by IMU and RGB-D fusion

[0160] As mentioned above, a real-time 3D modeling method for buildings driven by the fusion of IMU and RGB-D is proposed. The core idea of ​​this method is to use IMU data to estimate camera motion and adjust the sampling rate of the RGB-D camera according to the motion state, thereby significantly reducing redundant data collection, ensuring reconstruction quality while improving data transmission efficiency. Through IMU data prediction, dynamic targets are eliminated and SLAM pose solution is optimized; and based on the replacement matching fusion of point clouds between frames, combined with structural feature constraints, the 3D modeling accuracy is improved. Finally, the 3D modeling efficiency is improved through a progressive 3D modeling method.

[0161] Based on this method, the following system is finally integrated and designed:

[0162] Collection end:

[0163] The sampling rate is adjusted according to the IMU state. The sampling rate is reduced when stationary, maintained or slightly increased when moving steadily, and increased during intense exercise.

[0164] Server side:

[0165] Receive IMU and RGB-D raw data from sensors through Apache, and use PHP scripts for data transmission. Use algorithms for cluster analysis, and use the difference between the center of gravity coordinates of the predicted feature point cluster assisted by IMU and the actual feature point cluster, the dynamic feature perception method of center of gravity prediction and deviation detection to eliminate dynamic targets and optimize SLAM pose solution. Perform point cloud transformation and alignment according to the camera pose, and then compare and replace the point cloud accuracy between frames (the point cloud accuracy is determined by multi-index accuracy evaluation), optimize point cloud fusion in combination with feature constraints, and use Poisson estimation method to generate a mesh model, that is, progressive 3D modeling of outdoor scenes with real textures.

[0166] In terms of data management, the system uses the MySQL local database for data storage and management, and designs a reasonable data table structure to store IMU and RGB-D camera data, processed point cloud data, and mesh model data. For each point cloud, the coordinates of the camera pose when the RGB-D camera acquires the first image after the sampling rate is adjusted are converted to longitude and latitude coordinates, and then converted to Unity coordinates. The longitude is calculated as arctan 2(Y,X), and the latitude is The altitude is adjusted according to the earth model. The latitude change corresponds to the X-axis of Unity, using a conversion factor of 1 meter equal to approximately 0.00000899 degrees; the longitude change corresponds to the Z-axis of Unity, using a conversion factor at a specific latitude (such as 25°, 1 meter equals approximately 0.00000992 degrees); the altitude directly corresponds to the Y-axis of Unity, ensuring that 1 unit in Unity is equal to 1 meter, achieving precise spatial positioning. Finally, visualization is achieved in Unity, and the 3D model stored in the database is dynamically displayed.

[0167] In terms of system maintenance, we provide technical support and maintenance to ensure the stable operation of the system, and provide a friendly user interface and interactive functions to facilitate user use and operation.

[0168] Client:

[0169] Visualize and view the progressive 3D modeling of outdoor scenes built on the server. Relying mainly on accurate point cloud coordinates, the system ultimately has the functions of "measurable, analyzable, and computable". With these high-precision data, the system can accurately measure the size and distance of objects and environments, and recognize complex shapes and patterns, thereby supporting in-depth analysis. The detailed 3D information of the point cloud also makes computing tasks such as spatial coordinate conversion, path planning, and model generation more efficient and reliable.

[0170] In summary, this case proposes a real-time 3D modeling method and system for buildings driven by IMU and RGB-D fusion to achieve efficient and accurate pose estimation and 3D reconstruction. First, in the process of data acquisition, the motion state of the device is estimated in real time by analyzing the acceleration, angular velocity and other data provided by the IMU, and the sampling rate of the RGB-D camera is adaptively adjusted according to the motion state. When the device moves quickly, the sampling rate is increased to ensure that more environmental details are captured; when the device moves slowly or is stationary, the sampling rate is reduced to reduce redundant data, thereby improving data processing efficiency and reconstruction accuracy. Secondly, in the process of data processing, an IMU-assisted dynamic feature perception pose optimization algorithm is proposed. Through dynamic feature perception and elimination, pose correction is achieved to improve the accuracy of 3D reconstruction. Finally, in the 3D reconstruction process, point accuracy is evaluated by parameters such as depth, reprojection error, and point density, and the point cloud data with lower accuracy is replaced. Feature matching and boundary fusion are performed in combination with the structural features of the object to ensure the consistency and accuracy of the model, thereby achieving progressive 3D reconstruction of buildings.

Claims

1. A real-time three-dimensional modeling method for buildings driven by IMU and RGB-D fusion, characterized in that: The steps include: Step 1: Collect RGB-D camera data in real time based on outdoor scenes, including IMU data collected by the IMU sensor, depth images, color images and camera pose collected by the camera; Step 2: dynamically adjust the sampling rate of the RGB-D camera based on the adaptive sampling rate adjustment algorithm of the IMU data, and based on the adjusted sampling rate, the RGB-D camera scans the building to obtain the data of the RGB-D camera; Step 3: Based on the result obtained in step 2 and the dynamic feature perception pose solution optimization algorithm, the dynamic feature points are identified and eliminated to obtain the static feature pose of the RGB-D camera when each frame of depth image and color image is collected after optimization; Step 4: Based on the static feature pose and comprehensive accuracy evaluation of the point cloud of the RGB-D camera, the point clouds corresponding to the depth image and the color image are progressively replaced, and the progressive 3D modeling of the outdoor scene with realistic texture is performed in combination with the structural feature constraints.

2. The real-time three-dimensional modeling method of a building structure driven by IMU and RGB-D fusion according to claim 1 is characterized in that: The specific steps of step 2 are: Step 2.1, based on the IMU data collected in real time by the IMU sensor within a given time, the maximum deviation ratio among the deviation ratios is selected and compared with the corresponding threshold value, wherein the data collected by the IMU sensor includes acceleration and angular velocity, and the corresponding threshold value includes a stationary threshold value and a moving threshold value; The deviation ratio of the IMU data collected by the IMU sensor in real time within a given time includes the acceleration deviation ratio and the angular velocity deviation ratio; The acceleration deviation ratio formula is: The angular velocity deviation ratio formula is: In the formula, A t and W t They represent the acceleration deviation ratio and angular velocity deviation ratio at time t, respectively. a(t) and ω(t) are the acceleration and angular velocity at time t collected in real time. t and ω t For the acceleration threshold and the angular velocity threshold, the maximum deviation ratio is obtained from the acceleration deviation ratio and the angular velocity deviation ratio within a given time; Step 2.2, based on the result of comparing the maximum deviation ratio with the corresponding threshold, the motion state of the RGB-D camera when scanning the building is estimated in real time, where the motion state includes stillness, gentle movement or violent movement; The steps to estimate the motion state of the RGB-D camera when scanning a building in real time are: If the maximum deviation ratio is less than or equal to the static threshold, the motion state of the RGB-D camera when scanning the building is static, and the sampling rate needs to be reduced. When the RGB-D camera is always in a static state, the lower limit of the sampling rate reduction is f0 is the basic sampling rate; If the maximum deviation ratio is less than or equal to the motion threshold, the motion state of the RGB-D camera when scanning the building is smooth movement, and the sampling rate needs to be maintained at the basic sampling rate f0; If the maximum deviation ratio is greater than the motion threshold, the motion state of the RGB-D camera when scanning the building is violent motion, and the sampling rate needs to be increased. When the RGB-D camera is in a violent motion state, the upper limit of the sampling rate is f max =2f0; Step 2.3, dynamically adjusting the sampling rate of the RGB-D camera based on the motion state and the adaptive sampling rate adjustment algorithm based on the IMU data, and based on the adjusted sampling rate, the RGB-D camera starts scanning the building to obtain the data of the RGB-D camera; The formula for the adaptive sampling rate adjustment algorithm based on IMU data is: where f is the real-time sampling rate, i.e., the adjusted sampling rate. When the motion state is the stationary state, f min ≤ f < f0. When the motion state is intense exercise, f0 < f ≤ f max , k a , k ω are the adjustment coefficients of acceleration and angular velocity, and σ a represents the standard deviation of acceleration within a given time, and σ ω represents the standard deviation of angular velocity within a given time.

3. The real-time three-dimensional modeling method of a building structure driven by IMU and RGB-D fusion according to claim 2 is characterized in that: The specific steps of step 3 are: Step 3.1: Based on the data of the RGB-D camera obtained after the sampling rate is adjusted, the ORB algorithm is used to extract the ORB feature points of the Nth frame depth image or color image, and the density-based spatial clustering algorithm DBSCAN is used to generate the feature point cluster C for the extracted ORB feature points. i , where N ≥ 1; Step 3.2: According to each feature point cluster C in the Nth frame depth image or color image i The average value of all feature point coordinates in the corresponding feature point cluster C is obtained i The center of gravity coordinates G i , and based on the frame feature point cluster C i The center of gravity coordinates G i The IMU data collected in real time and adjusted by the sampling rate is used to predict the centroid coordinates of the feature point cluster with the same name in the N+1th frame depth image or color image. The prediction formula is as follows: In the formula, is the predicted centroid coordinate of the feature point cluster with the same name in the N+1th frame depth image or color image, R is the first integration result of the gyroscope data recorded by the IMU sensor within the frame interval, (Δx, Δy, Δz) is the second integration result of the linear acceleration recorded by the IMU sensor within the frame interval, G i is the centroid coordinate of the i-th feature point cluster; Step 3.3, use the ORB algorithm to extract the ORB feature points of the Nth frame depth image or color image, and use the fast neighbor algorithm to match the ORB feature points extracted from the N+1th frame depth image or color image with the ORB feature points extracted from the Nth frame. The successfully matched ORB feature points are marked as pairs of points with the same name. If the ORB feature points of the Nth frame in the pair of points with the same name belong to the feature point cluster of the Nth frame, Then the feature point of the N+1th frame in the same-name point pair belongs to the feature point cluster of the N+1th frame That is, the feature point cluster with the same name in the N+1th frame relative to the Nth frame. Finally, by calculating the feature point cluster with the same name The average value of all point coordinates The centroid coordinates of the measured feature point cluster with the same name Step 3.4, calculate the centroid coordinate deviation value ε between the measured feature point cluster with the same name and the predicted feature point cluster with the same name in the N+1th frame depth image or color image, the formula is: Step 3.5, if the centroid coordinate deviation value ε is greater than the set threshold, the cluster is considered to be a dynamic feature point cluster, and all ORB feature points in the cluster are removed; otherwise, the cluster is considered to be a static feature point cluster, and all ORB feature points in the cluster are retained; Step 3.6, based on the ORB feature points in the static feature point cluster of the N+1 frame obtained in step 3.5 and the ORB feature points in the feature point cluster with the same name corresponding to the Nth frame, form a static feature point pair. According to the static feature point pair, the initial static feature pose of the RGB-D camera when collecting depth images and color images is estimated by the PnP algorithm, and then the camera pose after the sampling rate adjustment is further optimized using nonlinearity to update the camera pose of the RGB-D camera after the sampling rate adjustment, until the above steps are completed for each frame, that is, the static feature pose of the RGB-D camera when collecting each depth image and color image is obtained based on the camera pose when taking the first frame of depth image and color image.

4. The real-time three-dimensional modeling method of a building structure driven by IMU and RGB-D fusion according to claim 3 is characterized in that: The specific steps of step 4 are: Step 4.1: Progressive point cloud replacement Based on the static feature pose, each pixel in the depth image or / and the color image is converted into a preliminary three-dimensional point cloud through the camera intrinsic rotation matrix, and the comprehensive accuracy of the point cloud of each frame of the preliminary three-dimensional point cloud is evaluated to perform accuracy partitioning, that is, for the Nth frame, by evaluating the comprehensive accuracy of the point cloud of each preliminary three-dimensional point cloud in the Nth frame, the point whose comprehensive accuracy of the point cloud is lower than the threshold is used as a replacement target, and the replacement target is marked after it is found, and the corresponding preliminary three-dimensional point cloud in the N+1 frame is found through the feature matching technology, so that the point clouds of two adjacent frames are accurately aligned, and the comprehensive accuracy of the point cloud of the preliminary three-dimensional point cloud corresponding to the N+1 frame is evaluated, and the comprehensive accuracy of the point cloud of the replacement target and its corresponding point in the adjacent frames is compared between frames. If the comprehensive accuracy of the point cloud of the N+1th frame is lower than that of the Nth frame, the point in the Nth frame is deleted, and if the comprehensive accuracy of the point cloud of the N+1th frame is higher than that of the Nth frame, the corresponding preliminary three-dimensional point cloud in the N+1th frame is used as the replacement point, and the attribute data of the replacement point in the N+1 frame is used as the attribute information of the replacement target in the Nth frame to complete the replacement, wherein the feature matching technology includes the ICP algorithm; The point cloud comprehensive accuracy evaluation formula of each preliminary 3D point cloud is: Among them, S m represents the comprehensive accuracy evaluation value of each preliminary 3D point cloud converted from the m-th frame depth image or / and color image, E m is the reprojection error formula, that is, E m =‖P(x m ,y m ,z m )-(u m ,v m )‖, where P is the projection matrix, (x m ,y m ,z m ) Preliminary 3D point cloud coordinates, (u m ,v m ) Two-dimensional point cloud projection coordinates, N m is the average neighbor distance method, Calculate the average distance between point m and its neighboring point j as the point density accuracy, D m is the depth accuracy of the mth point, d m is the depth value of the mth point, is the depth change rate of the mth point, let is the depth value d between the mth point and its neighboring point j j The average difference of , w1, w2 and w3 are weight parameters used to measure the importance of depth accuracy, reprojection error and point density in the comprehensive score, k represents the total number of preliminary 3D point clouds in the Nth frame, K represents the total variable of the preliminary 3D point cloud in the Nth frame; Step 4.2: Generate a mesh model based on the result obtained in step 4.1, the point cloud fusion optimization method with structural feature constraints and the Poisson estimation method, that is, perform progressive 3D modeling of outdoor scenes with real textures. The specific steps are as follows: According to the characteristics of buildings and structures, the basic data of multi-source buildings and structures are pre-integrated and stored based on prior knowledge and expert guidance to obtain a feature library. The structural features in the feature library include plumbness, edge-corner point, topology-orientation and continuity; Taking the point cloud corresponding to the depth image and color image after substitution optimization as the data basis, the structured data is parsed under the constraint of the feature library, that is, the principal component analysis method is used to identify and extract the structured point cloud data in the data basis that can match the features in the feature library. After extraction, the principal component analysis method is used to extract the same geometric structure in two adjacent frames, and then the feature matching method is used to match the same geometric structure features between two adjacent frames, so as to obtain the feature point pairs between frames; The rotation and translation matrices are calculated based on the feature point pairs between frames, and the point clouds of adjacent frames are aligned based on the rotation and translation matrices. After alignment, the ICP algorithm is used to iteratively optimize the correspondence between the two point clouds. If the convergence conditions are met, the final fine alignment result is obtained. Otherwise, the rotation and translation matrices are calculated again to align the point clouds of adjacent frames. Based on the feature point pairs between frames obtained after fine alignment, point cloud fusion is performed by combining the voxel grid method and the nearest neighbor search to optimize the fusion of point clouds in the overlapping areas of adjacent frame spaces. At the same time, in the boundary area, based on the feature point pairs between frames obtained after fine alignment, the boundary recognition and surface fitting method or the Laplace smoothing method are used to optimize the point cloud boundary fusion in the smooth boundary area. After optimization, the Poisson estimation method is finally used to perform progressive 3D modeling of outdoor scenes with real textures.

5. A real-time 3D modeling system for buildings driven by IMU and RGB-D fusion, characterized in that: include: Collection end: Collect RGB-D camera data in real time based on outdoor scenes, including IMU data collected by IMU sensors, depth images, color images and camera poses collected by cameras; The sampling rate of the RGB-D camera is dynamically adjusted by an adaptive sampling rate adjustment algorithm based on IMU data, and based on the adjusted sampling rate, the RGB-D camera scans the building to obtain the data of the RGB-D camera; Server side: Based on the RGB-D camera scanning the building after adjusting the sampling rate to obtain the RGB-D camera data and the dynamic feature perception pose solution optimization algorithm to identify and eliminate dynamic feature points, the static feature pose of the RGB-D camera when collecting each frame of depth image and color image is obtained after optimization; Based on the static feature pose and point cloud comprehensive accuracy evaluation of the RGB-D camera, the point clouds corresponding to the depth image and color image are progressively replaced, and the progressive 3D modeling of outdoor scenes with realistic textures is performed in combination with structural feature constraints. Client: Visualize and view the progressive 3D modeling of outdoor scenes built on the server.

6. The real-time three-dimensional modeling system of a building structure driven by IMU and RGB-D fusion according to claim 5, characterized in that: The specific steps of dynamically adjusting the sampling rate of the RGB-D camera based on the adaptive sampling rate adjustment algorithm of the IMU data in the acquisition end, and scanning the building with the RGB-D camera based on the adjusted sampling rate to obtain the data of the RGB-D camera are as follows: Step 2.1, based on the IMU data collected in real time by the IMU sensor within a given time, the maximum deviation ratio among the deviation ratios is selected and compared with the corresponding threshold value, wherein the data collected by the IMU sensor includes acceleration and angular velocity, and the corresponding threshold value includes a stationary threshold value and a moving threshold value; The deviation ratio of the IMU data collected by the IMU sensor in real time within a given time includes the acceleration deviation ratio and the angular velocity deviation ratio; The acceleration deviation ratio formula is: The angular velocity deviation ratio formula is: In the formula, A t and W t They represent the acceleration deviation ratio and angular velocity deviation ratio at time t, respectively. a(t) and ω(t) are the acceleration and angular velocity at time t collected in real time. t and ω t For the acceleration threshold and the angular velocity threshold, the maximum deviation ratio is obtained from the acceleration deviation ratio and the angular velocity deviation ratio within a given time; Step 2.2: Based on the result of comparing the maximum deviation ratio with the corresponding threshold, the motion state of the RGB-D camera when scanning the building is estimated in real time, where the motion state includes stillness, gentle movement or violent movement. The steps of estimating the motion state of the RGB-D camera when scanning the building in real time are: If the maximum deviation ratio is less than or equal to the static threshold, the motion state of the RGB-D camera when scanning the building is static, and the sampling rate needs to be reduced. When the RGB-D camera is always in a static state, the lower limit of the sampling rate reduction is f0 is the basic sampling rate; If the maximum deviation ratio is less than or equal to the motion threshold, the motion state of the RGB-D camera when scanning the building is smooth movement, and the sampling rate needs to be maintained at the basic sampling rate f0; If the maximum deviation ratio is greater than the motion threshold, the motion state of the RGB-D camera when scanning the building is violent motion, and the sampling rate needs to be increased. When the RGB-D camera is in a violent motion state, the upper limit of the sampling rate is f max =2f0; Step 2.3: The sampling rate of the RGB-D camera is dynamically adjusted based on the motion state and the adaptive sampling rate adjustment algorithm based on the IMU data. Based on the adjusted sampling rate, the RGB-D camera starts scanning the building to obtain the data of the RGB-D camera. The formula of the adaptive sampling rate adjustment algorithm based on the IMU data is: where f is the real-time sampling rate, i.e., the adjusted sampling rate. When the motion state is the static state, f min ≤ f < f0. When the motion state is intense exercise, f0 < f ≤ f max , k a , k ω are the adjustment coefficients of acceleration and angular velocity, and σ a represents the standard deviation of acceleration within a given time, and σ ω represents the standard deviation of angular velocity within a given time.

7. The real-time three-dimensional modeling system of a building structure driven by IMU and RGB-D fusion according to claim 6, characterized in that: The specific steps of obtaining the data of the RGB-D camera by scanning the building after adjusting the sampling rate and the dynamic feature perception posture solution optimization algorithm in the server side to identify and eliminate dynamic feature points, and obtaining the static feature posture of the RGB-D camera when collecting each frame of depth image and color image after optimization are as follows: Step 3.1: Based on the data of the RGB-D camera obtained after the sampling rate is adjusted, the ORB algorithm is used to extract the ORB feature points of the Nth frame depth image or color image, and the density-based spatial clustering algorithm DBSCAN is used to generate the feature point cluster C for the extracted ORB feature points. i , where N ≥ 1; Step 3.2: According to each feature point cluster C in the Nth frame depth image or color image i The average value of all feature point coordinates in the corresponding feature point cluster C is obtained i The center of gravity coordinates G i , and based on the frame feature point cluster C i The center of gravity coordinates G i The IMU data collected in real time and adjusted by the sampling rate is used to predict the centroid coordinates of the feature point cluster with the same name in the N+1th frame depth image or color image. The prediction formula is as follows: In the formula, is the predicted centroid coordinate of the feature point cluster with the same name in the N+1th frame depth image or color image, R is the first integration result of the gyroscope data recorded by the IMU sensor within the frame interval, (Δx, Δy, Δz) is the second integration result of the linear acceleration recorded by the IMU sensor within the frame interval, G i is the centroid coordinate of the i-th feature point cluster; Step 3.3, use the ORB algorithm to extract the ORB feature points of the Nth frame depth image or color image, and use the fast neighbor algorithm to match the ORB feature points extracted from the N+1th frame depth image or color image with the ORB feature points extracted from the Nth frame. The successfully matched ORB feature points are marked as pairs of points with the same name. If the ORB feature points of the Nth frame in the pair of points with the same name belong to the feature point cluster of the Nth frame, Then the feature point of the N+1th frame in the same-name point pair belongs to the feature point cluster of the N+1th frame That is, the feature point cluster with the same name in the N+1th frame relative to the Nth frame. Finally, by calculating the feature point cluster with the same name The average value of all point coordinates The centroid coordinates of the measured feature point cluster with the same name Step 3.4, calculate the centroid coordinate deviation value ε between the measured feature point cluster with the same name and the predicted feature point cluster with the same name in the N+1th frame depth image or color image, the formula is: Step 3.5, if the centroid coordinate deviation value ε is greater than the set threshold, the cluster is considered to be a dynamic feature point cluster, and all ORB feature points in the cluster are removed; otherwise, the cluster is considered to be a static feature point cluster, and all ORB feature points in the cluster are retained; Step 3.6, based on the ORB feature points in the static feature point cluster of the N+1 frame obtained in step 3.5 and the ORB feature points in the feature point cluster with the same name corresponding to the Nth frame, form a static feature point pair. According to the static feature point pair, the initial static feature pose of the RGB-D camera when collecting depth images and color images is estimated by the PnP algorithm, and then the camera pose after the sampling rate adjustment is further optimized using nonlinearity to update the camera pose of the RGB-D camera after the sampling rate adjustment, until the above steps are completed for each frame, that is, the static feature pose of the RGB-D camera when collecting each depth image and color image is obtained based on the camera pose when taking the first frame of depth image and color image.

8. The real-time three-dimensional modeling system of a building structure driven by IMU and RGB-D fusion according to claim 7, characterized in that: The specific steps of progressively replacing the point clouds corresponding to the depth image and the color image based on the static feature pose of the RGB-D camera and the comprehensive accuracy evaluation of the point cloud in the server side, and progressively modeling the outdoor scene with real texture in combination with the structural feature constraints are as follows: Step 4.1: Progressive point cloud replacement Based on the static feature pose, each pixel in the depth image or / and the color image is converted into a preliminary three-dimensional point cloud through the camera intrinsic rotation matrix, and the comprehensive accuracy of the point cloud of each frame of the preliminary three-dimensional point cloud is evaluated to perform accuracy partitioning, that is, for the Nth frame, by evaluating the comprehensive accuracy of the point cloud of each preliminary three-dimensional point cloud in the Nth frame, the point whose comprehensive accuracy of the point cloud is lower than the threshold is used as a replacement target, and the replacement target is marked after it is found, and the corresponding preliminary three-dimensional point cloud in the N+1 frame is found through the feature matching technology, so that the point clouds of two adjacent frames are accurately aligned, and the comprehensive accuracy of the point cloud of the preliminary three-dimensional point cloud corresponding to the N+1 frame is evaluated, and the comprehensive accuracy of the point cloud of the replacement target and its corresponding point in the adjacent frames is compared between frames. If the comprehensive accuracy of the point cloud of the N+1th frame is lower than that of the Nth frame, the point in the Nth frame is deleted, and if the comprehensive accuracy of the point cloud of the N+1th frame is higher than that of the Nth frame, the corresponding preliminary three-dimensional point cloud in the N+1th frame is used as the replacement point, and the attribute data of the replacement point in the N+1 frame is used as the attribute information of the replacement target in the Nth frame to complete the replacement, wherein the feature matching technology includes the ICP algorithm; The point cloud comprehensive accuracy evaluation formula of each preliminary 3D point cloud is: Among them, S m represents the comprehensive accuracy evaluation value of each preliminary 3D point cloud converted from the m-th frame depth image or / and color image, E m is the reprojection error formula, that is, E m =‖P(x m ,y m ,z m )-(u m ,v m )‖, where P is the projection matrix, (x m ,y m ,z m ) Preliminary 3D point cloud coordinates, (u m ,v m ) Two-dimensional point cloud projection coordinates, N m is the average neighbor distance method, Calculate the average distance between point m and its neighboring point j as the point density accuracy, D m is the depth accuracy of the mth point, d m is the depth value of the mth point, is the depth change rate of the mth point, let is the depth value d between the mth point and its neighboring point j j The average difference of , w1, w2 and w3 are weight parameters used to measure the importance of depth accuracy, reprojection error and point density in the comprehensive score, k represents the total number of preliminary 3D point clouds in the Nth frame, K represents the total variable of the preliminary 3D point cloud in the Nth frame; Step 4.2: Generate a mesh model based on the result obtained in step 4.1, the point cloud fusion optimization method with structural feature constraints and the Poisson estimation method, that is, perform progressive 3D modeling of outdoor scenes with real textures. The specific steps are as follows: According to the characteristics of buildings and structures, the basic data of multi-source buildings and structures are pre-integrated and stored based on prior knowledge and expert guidance to obtain a feature library. The structural features in the feature library include plumbness, edge-corner point, topology-orientation and continuity; Taking the point cloud corresponding to the depth image and color image after substitution optimization as the data basis, the structured data is parsed under the constraint of the feature library, that is, the principal component analysis method is used to identify and extract the structured point cloud data in the data basis that can match the features in the feature library. After extraction, the principal component analysis method is used to extract the same geometric structure in two adjacent frames, and then the feature matching method is used to match the same geometric structure features between two adjacent frames, so as to obtain the feature point pairs between frames; The rotation and translation matrices are calculated based on the feature point pairs between frames, and the point clouds of adjacent frames are aligned based on the rotation and translation matrices. After alignment, the ICP algorithm is used to iteratively optimize the correspondence between the two point clouds. If the convergence conditions are met, the final fine alignment result is obtained. Otherwise, the rotation and translation matrices are calculated again to align the point clouds of adjacent frames. Based on the feature point pairs between frames obtained after fine alignment, point cloud fusion is performed by combining the voxel grid method and the nearest neighbor search to optimize the fusion of point clouds in the overlapping areas of adjacent frame spaces. At the same time, in the boundary area, based on the feature point pairs between frames obtained after fine alignment, the boundary recognition and surface fitting method or the Laplace smoothing method are used to optimize the point cloud boundary fusion in the smooth boundary area. After optimization, the Poisson estimation method is finally used to perform progressive 3D modeling of outdoor scenes with real textures.

Images