Real-time dynamic 3D modeling method and system based on a group of point cloud sensors

Abstract

The present invention discloses a real-time dynamic three-dimensional modeling method and system based on a group of point cloud sensors. The system deploys a group of movable data acquisition terminals equipped with point cloud sensors in the area, scans the three-dimensional structural information of the area from different perspectives, and transmits it wirelessly to the aggregation server; the aggregation server aggregates the point cloud sequences from different perspectives into a unified coordinate system in real time to form a dynamic point cloud model. The present invention designs an efficient preprocessing paradigm for the data acquisition terminal and a real-time aggregation algorithm for multi-perspective point clouds of the aggregation server. Specifically, the data acquisition terminal uses visual inertial odometry and pose interpolation to estimate inter-frame pose changes, and uses an outlier detection algorithm based on a depth map to remove outliers. After in-place cross precoding, the encoding algorithm generates streaming data; the aggregation server receives and decodes the streaming data, reconstructs the scanned point cloud sequence, and uses a real-time multi-perspective point cloud aggregation algorithm based on point cloud registration and pose increment to construct a global point cloud sequence.

Description

Technical Field

[0001] The present application relates to a dynamic three-dimensional modeling system, specifically a real-time dynamic three-dimensional modeling method and system based on a group of point cloud sensors. Background Art

[0002] Computer-aided 3D modeling uses algorithms to analyze 2D images or 3D scans of real-world objects to create a digital 3D model of the object. This typically involves steps such as data acquisition, point cloud construction, and model rendering. Depending on the modeling results, 3D modeling techniques can be categorized as either static or dynamic.

[0003] Current dynamic 3D modeling methods can be roughly divided into two categories: dynamic scene modeling techniques based on a number of fixed point cloud sensors, and human motion capture techniques based on a number of fixed or mobile point cloud sensors. These methods either require the point cloud sensors to be fixed in position, which indirectly limits the selection of modeling scenes and the scalability of the system, or rely on prior knowledge of the human body or other specific modeling objects, making them incapable of handling complex modeling scenarios. Consequently, none of these existing methods possess the ability to model dynamic 3D scenes for large-scale scenes.

[0004] After searching the prior art, it was found that Chinese patent document CN115205489A, published on October 18, 2022, discloses a 3D reconstruction method for large scenes, including: obtaining image data of the reconstruction target through an RGB image acquisition device and performing preprocessing; searching and matching the images, calculating the feature points of each image, and matching the feature points; calculating the camera pose corresponding to each image; obtaining a dense point cloud intermediate model of the scene based on the image and the corresponding camera pose; post-processing the 3D point cloud model to finally obtain a 3D reconstructed mesh model. However, compared with the present invention, this prior art can only achieve static model construction and requires a long post-processing time, which is difficult to meet the needs of urgent tasks. Summary of the Invention

[0005] In response to the above-mentioned shortcomings of the existing technology, the present invention proposes a real-time dynamic three-dimensional modeling method and system based on a group of point cloud sensors. Based on the ideas of global relative pose estimation and incremental estimation, it combines the point cloud registration algorithm with the visual inertial odometry calculation method, and solves the problem of real-time transmission of point cloud data under wireless links, realizing information aggregation of multiple movable point cloud sensors and three-dimensional modeling of dynamic scenes.

[0006] The present invention is achieved through the following technical solutions:

[0007] The present invention relates to a real-time dynamic 3D modeling method based on a group of point cloud sensors. Multiple movable point cloud sensors acquire 3D structural data of a regional scene from different perspectives. After processing using a fast outlier removal method, the data is sent to an aggregation server using coded transmission. The aggregation server obtains scanned point clouds and posture change data from multiple perspectives, aggregates them in real time into a unified coordinate system using an algorithm based on point cloud registration and posture increment estimation, forms a global dynamic point cloud model of the region, and updates the data to a display window in real time.

[0008] The present invention also relates to a system for implementing the above method, comprising a plurality of data collection terminals and an aggregation server.

[0009] The data acquisition terminal can move freely and is composed of a point cloud sensor (mechanical laser radar, Flash solid-state laser radar or depth camera), an integrated visible light camera, an integrated inertial measurement unit, a data transmission module and a mobile computer, wherein: the point cloud sensor uses the time of flight (ToF) method to measure the three-dimensional structure of the environment to obtain a depth map containing local three-dimensional information; the visible light camera records the RGB image of the scene; the inertial measurement unit outputs its own inertial measurement data (acceleration and angular velocity); the mobile computer collects the above data, and estimates the inter-frame pose change of the point cloud sensor based on the RGB image and inertial measurement data using the visual inertial odometry and pose interpolation method, and uses the fast outlier removal method and bit cross precoding to process the depth map; the data transmission module sends the depth map sequence, RGB image sequence and pose change data sequence to the aggregation server through encoded transmission.

[0010] The aggregation server includes a data receiving module, a point cloud construction module, a global point cloud aggregation module and a display module, wherein: the data receiving module receives and decodes streaming data from multiple data acquisition terminals to obtain corresponding depth map sequences, RGB image sequences and posture change data sequences; the point cloud construction module converts the depth map sequence and RGB image sequence into a scanned point cloud sequence according to the pre-calibrated internal parameters of the point cloud sensor; the global point cloud aggregation module utilizes a real-time aggregation algorithm based on point cloud registration and posture increment estimation to aggregate scanned point cloud sequences of different perspectives into a unique global point cloud sequence in real time; the display module displays the global point cloud sequence in the screen window of the aggregation server to form a global dynamic point cloud model.

[0011] Technical Effects

[0012] The technical means disclosed by the present invention, which have never been made public, include: a fast point cloud outlier detection algorithm based on depth map; a bit-interleaved precoding method for processing depth map into an 8-bit image format that can be processed by video coding algorithm; and a real-time multi-view point cloud aggregation algorithm based on point cloud registration and pose increment.

[0013] The technical effect brought about by the above-mentioned technical means, which has never been disclosed before, is that the depth map sequence, RGB image sequence and posture change data sequence obtained by the data acquisition end can be transmitted to the aggregation server in real time, and the aggregation server can aggregate the scanning point clouds from different perspectives into a unique global point cloud model in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Flowchart of the present invention;

[0015] Figure 2 This is an example diagram of the outlier removal algorithm based on the depth map;

[0016] Figure 3 Schematic diagram of depth map precoding with bit interleaving;

[0017] Figure 4 This is an example diagram of global point cloud aggregation. DETAILED DESCRIPTION

[0018] like Figure 1 As shown in FIG. 1 , a real-time dynamic 3D modeling method based on a group of point cloud sensors involved in this embodiment specifically includes:

[0019] Step 1) Deployment and data collection of data acquisition terminals. Multiple data acquisition terminals are deployed within the modeling area. They move around designated points of interest and scan the scene's 3D structure. On each data acquisition terminal, a mobile computer acquires sensor data from the point cloud sensor, visible light camera, and inertial measurement unit.

[0020] Step 2) Pose estimation based on visual inertial odometry and interpolation methods at the data acquisition end, specifically including:

[0021] 2.1) The mobile computer inputs the acquired RGB image sequence and inertial measurement data sequence into the visual-inertial odometry module. The visual-inertial odometry module only calculates and outputs motion estimation results for the key RGB frames, including the position vector p, velocity vector v, and attitude quaternion q. Different visual-inertial odometry algorithms have different keyframe selection strategies, but typically, the time between two adjacent keyframes is only a short one.

[0022] 2.2) Use the interpolation method to calculate the pose data of other image frames between adjacent key frames, specifically:

[0023] Assume that the timestamps of adjacent key frames are T i With T i+1 The timestamp of the non-key frame to be calculated is t(T i <t<T i+1 ).

[0024] 2.2.1) Using the cubic Hermite interpolation method, the position vector p(t) of the non-key frame is calculated based on the position and velocity of the adjacent key frames, specifically including:

[0025] Assume that the key frame position vector output by the visual inertial odometry module is p i With p i+1 , the velocity vector is v i With v i+1 . First calculate the interpolation basis function and Then the position vector of the non-key frame is p(t) = α0(t)·p i +α1(t)·p i+1 +β0(t)·v i +β1(t)·v i+1 , where t satisfies T i <t<T i+1 .

[0026] 2.2.2) Using spherical linear interpolation of quaternions, the pose estimate q(t) of the non-keyframe is calculated based on the poses of adjacent keyframes, specifically including:

[0027] Assume that the key frame attitude quaternion output by the visual inertial odometry module is q i With q i+1 If the attitude quaternion is The corresponding rotation matrix is The rotation matrices corresponding to the quaternions of the two key frames are R i With R i+1 , then the attitude quaternion of the frame time, that is, the attitude estimation is in q i With q i+1 Angle.

[0028] 2.3) Based on step 2.2, the pose estimates of all RGB frames are obtained. By aligning the timestamps of the depth map with the timestamps of the RGB frames, the pose estimates of the depth map can be obtained.

[0029] Step 3) Fast outlier removal based on depth map at the data acquisition end, specifically:

[0030] Due to the defects of the point cloud sensor itself, there are inevitably some sparse outliers in the point cloud data it collects that do not reflect the actual scene. These outliers are of no benefit to the processing and display of the point cloud, but will increase the space occupied by the point cloud data and the transmission bandwidth burden. In order to remove outliers in the point cloud, the commonly used method is to use an outlier detection algorithm based on statistical filtering or radius filtering for processing, but these methods take too long to execute on mobile computers and are difficult to meet the system's real-time requirements. Therefore, the present invention proposes a fast outlier removal algorithm based on a depth map, which specifically includes:

[0031] First, a scanned point cloud is completely projected onto a depth sensing plane (for Flash solid-state LiDAR or depth camera) or a cylindrical surface (for mechanical LiDAR) to form a two-dimensional depth frame, where each pixel represents the distance of the point relative to the sensor. Then, the algorithm removes points on the depth map plane whose ratio of surrounding valid pixels is less than a preset threshold, such as Figure 2 As shown, specifically: for a depth map of size H×W, a two-dimensional auxiliary matrix M of the same size is constructed in linear time, where: M x，y Represents the number of valid pixels in the sub-image with the origin as the upper left corner and the xth row and yth column as the lower right corner; using M, the number of valid pixels in the r-rectangular neighborhood (r is the side length of the matrix) of the xth row and yth column pixel can be calculated in constant time as S x，y =M x+r，y+r -M x，y+r -M x+r，y +M x-r，y-r -I x，y , where I x，y It is the valid indicator of the pixel in row x and column y. It is 1 when the pixel is valid, otherwise it is 0. When x±r or y±r exceeds the size range of M, the corresponding value is replaced by the boundary value, such as M 7，-2 M 7，0 replace.

[0032] This algorithm compresses the three-dimensional outlier detection problem into two-dimensional space, thereby greatly improving the execution efficiency and being able to process all data collected by point cloud sensors in real time on mobile computers.

[0033] Step 4) Real-time encoding and transmission of data at the data acquisition end, specifically:

[0034] Since the hardware encoding algorithm in the data transmission module usually does not accept image input of depth pixels, this embodiment designs the following Figure 3The bit-interleaved pre-coding shown in FIG2 decomposes the depth frame of the depth pixel into multiple 8-bit pixel frames and encodes them separately. Specifically, taking 16-bit depth pixels as an example, the even bits are placed in the first 8-bit pixel in descending order, and the odd bits are placed in the second 8-bit pixel in the same way, thereby splitting the 16-bit depth map into two 8-bit images, as shown in FIG3 . Figure 3 This method encodes each pixel individually and can process each depth map in linear time.

[0035] The system then delivers the split depth map sequence and RGB image sequence to the data transmission module. Preferably, this embodiment uses low-fidelity encoding settings to process the RGB image sequence to reduce bandwidth usage, while using high-fidelity encoding settings to process the depth map sequence to minimize point cloud reconstruction errors, that is, to reduce the difference between the decoded and reconstructed point cloud and the original point cloud. The pose change data is transmitted to the aggregation server in its raw byte representation in little-endian order.

[0036] Step 5) The aggregation server receives and decodes data and reconstructs the scanned point cloud from the depth map. Specifically, the aggregation server uses a data receiving module to receive and decode the encoded stream data from multiple data acquisition terminals, obtaining a split depth map sequence, an RGB image sequence, and a pose change data sequence. The split depth map sequence is then merged into a depth map using reverse bit-interleaved precoding. For each depth map, each valid pixel in the 2D depth map is converted into a point in a 3D coordinate system based on the specific type of point cloud sensor and the pre-calibrated sensor internal parameter values ​​to form a point cloud.

[0037] Step 6) Real-time multi-view point cloud aggregation on the aggregation server, specifically including:

[0038] 6.1) Assume that there are N data acquisition terminals in the system. The aggregation server will obtain N scan point cloud sequences. After aligning them according to the timestamps, a sequence of point cloud sets can be obtained, where each point cloud set includes N scan point clouds.

[0039] 6.2) In the initial state, for the N point clouds in the first point cloud set, the aggregation server runs a registration algorithm between each pair and selects the N-1 pairs of point cloud registration results with the highest registration overlap rate as the initial coordinate transformation estimate to obtain the relative positions of the N data acquisition terminals at the initial moment.

[0040] The registration algorithm is specifically the open-source feature extraction-based PREDATOR registration algorithm (Shengyu Huang, et al. "Predator: Registration of 3D point clouds with low overlap." Proceedings of the IEEE / CVF Conference on computer vision and pattern recognition. 2021.), which introduces an overlapping attention block into the feature extraction network to exchange information between the latent codes of the source point cloud and the target point cloud. Therefore, PREDATOR can discover salient points located in the overlapping area, thereby achieving higher registration accuracy.

[0041] 6.3) For the N point clouds in the point cloud set at the next moment in the sequence, the relative pose at the previous moment is quickly calculated in a pose increment manner based on the estimated pose changes of each data acquisition terminal. Until the preset or adaptive registration time limit, the aggregation server selects the N-1 pairs of point clouds with the highest field of view overlap at the previous moment and runs the registration algorithm on them to obtain the relative pose at that moment.

[0042] The posture increment method specifically means that when the aggregation server masters the relative posture relationship of all scanned point clouds in the point cloud set at the previous moment, and has the local estimation results of the posture changes between point cloud frames at each data acquisition end, it can obtain the relative posture relationship at the next moment by applying corresponding N posture changes to the relative posture relationship of N point clouds at the previous moment.

[0043] The adaptive registration time limit is determined by the time it takes to perform a round of registration. Specifically, let the timestamp of the aggregation server be T i A set of point clouds performs pose registration, and the total time consumption is t. Then the adaptive registration time limit of this round is also t, that is, the aggregation server immediately performs pose registration on the points with timestamps greater than T. i +t performs pose registration on the first set of point clouds, and performs pose registration on the points with timestamps at T in parallel. i With T i Incremental pose estimation is applied to several groups of point clouds between +t.

[0044] The visual angle overlap is calculated as follows: a regular pyramid is drawn with each data acquisition end position as the vertex. Its vertex angle is determined according to the sensor's visual angle range, and its height is the sensor's maximum ranging distance. The ratio of the overlapping volume of the two regular pyramids to the volume of a single pyramid is the visual angle overlap of the corresponding two data acquisition ends.

[0045] After obtaining the relative pose relationship of a set of point clouds, the aggregation server can convert the point clouds from different perspectives into a unified coordinate system through coordinate transformations of translation and rotation, thereby obtaining a global point cloud model of the scene state at that moment.

[0046] Figure 4 The figure shows the point cloud aggregation results from five data collection terminals at a certain moment. The arrows in the global point cloud model represent the sensor positions obtained using the real-time multi-view point cloud aggregation algorithm. The sequence of aggregated point clouds arranged by timestamps constitutes the dynamic point cloud model of the scene.

[0047] Step 7) Presenting the dynamic global 3D model: Specifically, after the above steps, the aggregation server generates a series of global point cloud models reflecting the real-time scene and updates them to the display window at the frequency of the point cloud sensor scans. Ultimately, the system user can browse this dynamic point cloud from a free perspective, thereby understanding the real-time status and changing trends of the scene environment.

[0048] After actual experiments, in a system setting where 5 data acquisition terminals and 1 aggregation server are deployed and each point cloud sensor operates at a scanning frequency of 30 frames per second, the above embodiment is run, and experimental data can be obtained: the aggregation server displays a global dynamic point cloud at 30 frames per second, the point cloud model reflects the three-dimensional structure of the deployed real scene, and the delay of the dynamic point cloud model changing with the real scene does not exceed 2 seconds; the aggregation server takes about 2.5 seconds to perform a round of alignment, so in every approximately 30 frames of global point cloud, one frame is constructed by the relative pose calculated by the registration algorithm, and the remaining 29 frames are constructed by the relative pose calculated by the pose increment method.

[0049] Compared with the existing technology, the performance indicators of this embodiment are improved in that it realizes real-time dynamic three-dimensional modeling based on multiple movable point cloud sensors, can build dynamic three-dimensional models of large-scale outdoor environments in real time, and has good scalability for the number of point cloud sensors.

[0050] In addition, the main difficulties solved by this embodiment include:

[0051] First, a scalable, real-time aggregation algorithm for multi-view point clouds was designed for the aggregation server. Because point cloud registration algorithms are time-consuming and difficult to execute in real time to process point cloud sensor data, this embodiment designed a real-time aggregation algorithm for multi-view point clouds that integrates point cloud registration with incremental pose estimation. This algorithm can process data from up to ten point cloud sensors.

[0052] Secondly, a real-time data acquisition and processing method is designed for data acquisition terminals with limited computing resources. For cost and flexibility reasons, data acquisition terminals are typically equipped with only mobile computers with limited computing power. This embodiment utilizes interpolation to improve the visual-inertial odometry implementation to enable real-time processing of sensor data.

[0053] Finally, real-time transmission of 3D sensing data is achieved over a bandwidth-limited wireless link. To maximize the mobility of the data acquisition endpoint, the system uses a wireless link to connect the data acquisition endpoint to the aggregation server. This embodiment designs an outlier removal algorithm based on the depth map to reduce the required transmission bandwidth, while also utilizing hardware encoding to achieve efficient, high-fidelity data stream encoding.

[0054] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principles and purpose of the present invention. The scope of protection of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. All implementation schemes within its scope shall be subject to the constraints of the present invention.

Claims

1. A real-time dynamic 3D modeling method based on a set of point cloud sensors, characterized in that: The method comprises: Within the modeling area, a group of mobile data acquisition terminals equipped with point cloud sensors are deployed to continuously scan the three-dimensional information of the environment from different perspectives, namely, depth map sequences; Each data acquisition terminal simultaneously uses an integrated visible light camera to capture RGB images of the scene and collects inertial measurement data from an integrated inertial measurement unit for pose change estimation based on visual inertial odometry and interpolation methods; The mobile computing platform at the data collection end uses a fast point cloud outlier removal algorithm based on depth maps to process each depth map in real time, and uses a bit-interleaved depth map pre-coding method to process it and convert it into an 8-bit grayscale image acceptable to the hardware encoding algorithm; The data collection end transmits the encoded depth stream data, RGB stream data, and pose change data to an aggregation server in real time via a wireless link; The aggregation server decodes the depth stream to obtain a depth map sequence and an RGB image sequence, and reconstructs the scanned point cloud sequence using the pre-calibrated point cloud sensor internal parameters, and has an estimate of the pose change of each frame of the scanned point cloud; The aggregation server uses timestamps to align point cloud sequences from different data acquisition terminals to obtain a point cloud set sequence. Each point cloud set contains scanned point clouds from different data acquisition terminals at similar times. For this point cloud sequence, the aggregation server uses a real-time multi-view point cloud aggregation algorithm based on point cloud registration and pose increment to construct a global point cloud sequence; The aggregation server obtains the relative poses of all point clouds in each point cloud set, and can transform these point clouds into a unified coordinate system to form a global point cloud of the scene at that moment; A series of global point clouds constructed using a point cloud set sequence can be presented to users in the form of a dynamic point cloud model.

2. The method according to claim 1, characterized in that The pose change estimation based on visual inertial odometry and interpolation method includes: The data acquisition end inputs the inertial measurement data of the RGB image frame sequence into the visual inertial odometry module to obtain the pose estimation of the key RGB frame; The pose of non-key RGB image frames is calculated based on the pose estimation results of the previous and next key frames using the interpolation method. The position vector is calculated using cubic Hermite interpolation, and the attitude quaternion is calculated using spherical linear interpolation. By aligning the timestamps of the RGB image and the scanned point cloud, the pose estimation result of the scanned point cloud can be obtained.

3. The method according to claim 1, characterized in that The fast point cloud outlier removal algorithm based on depth map includes: For each pixel in the depth map, the data acquisition end calculates the ratio of valid points within the surrounding rectangular range and considers pixels with a valid point ratio less than a preset threshold as outliers. For sizes The depth map can be constructed in linear time with the auxiliary matrix of the same size. Rank The elements of the column store the number of valid pixels in the sub-image with the origin as the upper left corner and the origin as the lower right corner. The number of valid pixels around any pixel can be obtained in constant time through the auxiliary matrix. Therefore, the algorithm can remove outliers for a single depth map in linear time.

4. The method according to claim 1, wherein The bit-interleaved depth map precoding method includes: Depth map precoding converts depth pixels in the depth map into multiple 8-bit pixels to form multiple 8-bit images for easy encoding; The even-numbered bits are placed in the first 8-bit pixel in descending order, while the odd-numbered bits are placed in the second 8-bit pixel in the same way, thereby splitting the 16-bit depth map into two 8-bit images.

5. The method according to claim 1, wherein The real-time multi-view point cloud aggregation algorithm based on point cloud registration and pose increment includes: For the initial point cloud set, the aggregation server applies the registration algorithm to all the point cloud pairs and calculates the overlap rate. The set of pose registration results with the highest overlap rate is selected as the initial relative pose estimate. Subsequently, the aggregation server uses the relative pose estimate at the previous moment and the pose change estimate between frames to quickly calculate the relative pose at the next moment; In addition, the aggregation server periodically or adaptively runs a registration algorithm to correct the relative pose estimates to avoid severe deviations in the global point cloud.

6. An intelligent point cloud sensor as a data acquisition terminal, characterized in that: The equipment has three-dimensional sensing, computing, and communication capabilities, including: The device includes a point cloud sensor, an integrated visible light camera, an integrated inertial measurement unit, a mobile computer, a wireless communication module, and a battery; The device is based on a mobile computer and is connected to other device modules so that it can work as a complete system to implement the method according to any one of claims 1 to 5.

Images