Three-dimensional point cloud reconstruction method and device

Abstract

The application discloses a three-dimensional point cloud reconstruction method and device, and relates to the technical field of computer vision three-dimensional reconstruction, and the method comprises the following steps: acquiring RGB images of a plurality of RGB cameras and depth maps corresponding to the RGB images; performing pixel back-projection on the RGB images and the depth maps based on a pinhole camera geometric model to obtain a first three-dimensional point cloud of each RGB camera; performing registration on a plurality of first three-dimensional point clouds based on pre-acquired external parameter information of the plurality of RGB cameras to obtain a plurality of second three-dimensional point clouds after registration; and performing spatial down-sampling fusion on the plurality of second three-dimensional point clouds based on hash mapping to obtain a fused three-dimensional point cloud. The application does not require a complex neural network model, can reduce the calculation complexity of three-dimensional point cloud reconstruction, and realizes real-time point cloud reconstruction.

Description

Technical Field

[0001] This invention relates to the field of computer vision 3D reconstruction technology, and in particular to a 3D point cloud reconstruction method and apparatus. Background Technology

[0002] 3D point cloud reconstruction refers to the process of recovering a large number of discrete 3D coordinate points of a real object / scene in 3D space using 2D RGB images and depth maps, and then using these 3D coordinate points to reconstruct the 3D shape of the object. 3D point cloud reconstruction technology has been widely used in computer vision, robot perception, digital twins, and other fields.

[0003] Existing 3D point cloud reconstruction methods rely on complex neural network models for computation, resulting in high computational complexity and difficulty in meeting real-time requirements. Summary of the Invention

[0004] This invention provides a method for reconstructing three-dimensional point clouds without requiring complex neural network models, thereby reducing the computational complexity of three-dimensional point cloud reconstruction and enabling real-time point cloud reconstruction. The method includes: Acquire RGB images from multiple RGB cameras and depth maps corresponding to the RGB images; Based on the geometric model of the pinhole camera, pixel back projection is performed on the RGB image and depth map to obtain the first three-dimensional point cloud of each RGB camera; Based on the pre-acquired extrinsic information of multiple RGB cameras, multiple first 3D point clouds are registered to obtain multiple registered second 3D point clouds. Multiple second-dimensional point clouds are spatially downsampled and fused based on hash mapping to obtain a fused 3D point cloud.

[0005] This invention also provides a 3D point cloud reconstruction device that eliminates the need for complex neural network models, reducing the computational complexity of 3D point cloud reconstruction and enabling real-time point cloud reconstruction. The device includes: An image acquisition module is used to acquire RGB images from multiple RGB cameras and depth maps corresponding to the RGB images; The back projection module is used to perform pixel back projection on the RGB image and depth map based on the geometric model of the pinhole camera to obtain the first three-dimensional point cloud of each RGB camera. The registration module is used to register multiple first 3D point clouds based on multiple pre-acquired RGB camera extrinsic information to obtain multiple registered second 3D point clouds. The fusion module is used to perform spatial downsampling fusion of multiple second-dimensional point clouds based on hash mapping to obtain a fused 3D point cloud.

[0006] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described three-dimensional point cloud reconstruction method.

[0007] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described three-dimensional point cloud reconstruction method.

[0008] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described three-dimensional point cloud reconstruction method.

[0009] In this embodiment of the invention, RGB images from multiple RGB cameras and corresponding depth maps are acquired. Based on the geometric model of the pinhole camera, pixel back-projection is performed on the RGB images and depth maps to obtain a first 3D point cloud for each RGB camera. Based on pre-acquired extrinsic information from multiple RGB cameras, the multiple first 3D point clouds are registered to obtain multiple registered second 3D point clouds. Spatial downsampling and fusion of the multiple second 3D point clouds are performed based on hash mapping to obtain a fused 3D point cloud. This method does not require a complex neural network model. Back-projection based on the geometric model of the pinhole camera allows all RGB pixels to be mapped to 3D points using only linear arithmetic operations, reducing computational complexity. Furthermore, spatial downsampling and fusion of the multiple second 3D point clouds based on hash mapping can further reduce computational complexity while ensuring point cloud accuracy, thereby achieving real-time point cloud reconstruction. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] In the attached diagram: Figure 1 This is a flowchart of a three-dimensional point cloud reconstruction method provided in an embodiment of the present invention; Figure 2 This is a flowchart of another three-dimensional point cloud reconstruction method provided in an embodiment of the present invention; Figure 3 This is a flowchart of another three-dimensional point cloud reconstruction method provided in an embodiment of the present invention; Figure 4This is a flowchart of a method for spatial downsampling and fusing multiple second three-dimensional point clouds based on hash mapping to obtain a fused three-dimensional point cloud, provided in an embodiment of the present invention. Figure 5 This is a flowchart of another method for spatial downsampling and fusing multiple second three-dimensional point clouds based on hash mapping to obtain a fused three-dimensional point cloud, provided in an embodiment of the present invention. Figure 6 This is a schematic diagram of a three-dimensional point cloud reconstruction device provided in an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0013] In the description of this specification, the terms "comprising," "including," "having," and "containing" are open-ended terms, meaning that they include but are not limited to. The terms "an embodiment," "a specific embodiment," "some embodiments," and "for example," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example that are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. The order of steps involved in the various embodiments is used to illustrate the implementation of this application, and the order of steps is not limited and can be adjusted appropriately as needed.

[0014] Research has revealed the following problems with existing 3D point cloud reconstruction methods: 1. Existing 3D point cloud reconstruction methods rely on complex neural network models for computation, which has high computational complexity and is difficult to meet real-time requirements.

[0015] 2. In multi-camera acquisition scenarios, due to differences in exposure parameters and lighting conditions of each camera, the point cloud colors in different views have brightness differences, and direct fusion will lead to inconsistent colors.

[0016] 3. When fusing point clouds from multiple views, the overlapping areas observed by different cameras will generate a large number of redundant points, resulting in low computational efficiency and requiring a lot of computing resources.

[0017] 4. The optical center positions of the depth camera and the RGB camera are different, resulting in a pixel-level offset between the depth map and the RGB image, which leads to lower accuracy in 3D reconstruction.

[0018] Based on this, embodiments of the present invention provide a three-dimensional point cloud reconstruction scheme to solve the above problems.

[0019] Example 1 Figure 1 A flowchart of a three-dimensional point cloud reconstruction method provided in an embodiment of the present invention, the method comprising: Step 101: Acquire RGB images from multiple RGB cameras and depth maps corresponding to the RGB images; Step 102: Based on the geometric model of the pinhole camera, perform pixel back projection on the RGB image and depth map to obtain the first three-dimensional point cloud of each RGB camera; Step 103: Based on the pre-acquired extrinsic information of multiple RGB cameras, register multiple first 3D point clouds to obtain multiple registered second 3D point clouds; Step 104: Spatial downsampling and fusion of multiple second 3D point clouds based on hash mapping to obtain the fused 3D point cloud.

[0020] In step 101 above, RGB images from multiple RGB cameras and depth maps corresponding to the RGB images are acquired.

[0021] In practice, multiple RGB cameras are calibrated RGB-D cameras (i.e., a combination of RGB cameras and depth cameras), and each RGB-D camera can output RGB images and depth maps simultaneously.

[0022] Alternatively, multiple RGB cameras can be configured to be calibrated RGB cameras, where each RGB camera only outputs an RGB image. The depth map corresponding to the RGB image can be obtained through a separate depth camera, or it can be obtained based on the RGB image using a depth estimation neural network (such as a monocular depth estimation network or a binocular stereo matching algorithm). Since the methods for obtaining the depth map are all existing mature technologies, they will not be elaborated on here.

[0023] It should be noted that multiple RGB cameras are deployed in different locations to acquire RGB images of the same object from different perspectives, as well as depth maps corresponding to the RGB images.

[0024] If the depth map corresponding to the RGB image is acquired using a depth camera, such as an RGB-D camera, or a separate depth camera, the difference in optical center position or resolution between the depth camera and the RGB camera leads to pixel offset between the depth map and the RGB image. Therefore, pixel alignment of the depth map is required (aligning each RGB pixel in the RGB image with the depth value in the depth map), i.e., executing Example 2. If the depth map corresponding to the RGB image is predicted by a depth estimation network, the predicted depth map naturally corresponds pixel-by-pixel to the input RGB image, and pixel alignment of the depth map is not required, i.e., Example 2 is not required.

[0025] After acquiring RGB images from multiple RGB cameras and the corresponding depth maps, the process may include at least one of the following preprocessing steps on the RGB images and depth maps: depth scale normalization, adaptive impulse noise removal, adaptive erosion of near edges, and layered hole filling of distant views.

[0026] Among them, depth scale normalization can be based on a preset effective depth range [minimum depth value, maximum depth value], and invalidate depth pixels in the depth map whose depth values ​​do not meet the effective depth range.

[0027] Adaptive impulse noise removal can be achieved by setting a differential threshold based on the distance between RGB pixels in an RGB image, and filtering out isolated flying points through mean convolution and the proportion of effective neighborhoods.

[0028] Near-field adaptive erosion can perform morphological processing only on nearby isolated abrupt edges in an RGB image, preserving the continuous object contour.

[0029] Layered hole filling in the distant view can be done by applying a weighted average diffusion filling method only to distant holes beyond 2m (i.e., holes with a depth value beyond 2m), while leaving the foreground within 2m unfilled to avoid foreground distortion.

[0030] It should be noted that in step 101, the RGB image and depth map can be transferred to the video memory of the graphics processing unit (GPU) via a direct memory access (DMA) channel. This allows the 3D point cloud reconstruction method to be executed on the GPU, eliminating CPU-GPU bandwidth loss and resolving frame drop issues in multi-device synchronous acquisition.

[0031] In step 102 above, based on the geometric model of the pinhole camera, pixel back projection is performed on the RGB image and depth map to obtain the first three-dimensional point cloud of each RGB camera.

[0032] In practical implementation, the geometric model of a pinhole camera can be represented by the following formula: in, Represents the pixel coordinates of an RGB pixel. , For the pre-acquired RGB camera focal length, For the pre-acquired optical center coordinates of the RGB camera, The depth value corresponding to the RGB pixel. These are the coordinates of a 3D point obtained after pixel backprojection.

[0033] Based on the formula of the above pinhole camera geometric model, the RGB pixels in each RGB image can be... and RGB pixels in the depth map The corresponding depth value Z is mapped to three-dimensional coordinates, i.e., back-projected pixel by pixel to obtain the first three-dimensional point cloud for each RGB camera. Each first three-dimensional point cloud includes at least three-dimensional points. In addition to the coordinates, it can also include three-dimensional points. The corresponding RGB values ​​can be obtained from the RGB image.

[0034] In practice, since there are differences in brightness and exposure in the RGB images captured by different RGB cameras, after step 102 above, the RGB values ​​in the first three-dimensional point cloud can be corrected, that is, the third embodiment is executed, in order to eliminate the differences in brightness and exposure caused by the RGB camera acquisition and further improve the accuracy of point cloud reconstruction.

[0035] In step 103 above, based on the pre-acquired extrinsic information of multiple RGB cameras, multiple first three-dimensional point clouds are registered to obtain multiple registered second three-dimensional point clouds.

[0036] In practice, the extrinsic information of each RGB camera can include a rotation matrix R and a translation vector t. During system initialization, the extrinsic information of all RGB cameras can be loaded into the GPU constant memory. The extrinsic information of multiple RGB cameras is pre-calibrated extrinsic information. The specific calibration method is not limited here.

[0037] It can analyze 3D points in the first 3D point cloud. Perform the following transformations in batches: in, The coordinates of the registered 3D points are shown below. This allows us to obtain a second 3D point cloud in a unified world coordinate system, avoiding the problem of misaligned and layered point clouds caused by inconsistent camera coordinate systems of different RGB cameras. Moreover, step 103, which registers the point cloud based on the extrinsic parameters of the RGB cameras, only uses linear matrix operations, without iterative optimization or nonlinear solutions, and is fully compatible with the full GPU vectorized parallel pipeline, without introducing additional computational overhead.

[0038] In step 104 above, multiple second three-dimensional point clouds are spatially downsampled and fused based on hash mapping to obtain the fused three-dimensional point cloud.

[0039] In practical implementation, when fusing the second and third-dimensional point clouds, a tiered hashing deduplication scheme can be adopted, using different downsampling strategies for point clouds at different depths to balance point cloud accuracy and memory usage. Specifically, multiple second and third-dimensional point clouds can be merged first, and then the merged second and third-dimensional point clouds can be divided into point cloud sets of different depths according to a preset depth segmentation threshold. Downsampling at different scales can then be performed on point cloud sets of different depths.

[0040] In one embodiment, step 104 can be implemented through embodiment four or embodiment five.

[0041] Based on this, the aforementioned 3D point cloud reconstruction method acquires RGB images from multiple RGB cameras and corresponding depth maps; performs pixel back-projection on the RGB images and depth maps based on the pinhole camera's geometric model to obtain a first 3D point cloud for each RGB camera; registers the multiple first 3D point clouds based on pre-acquired extrinsic information from multiple RGB cameras to obtain multiple registered second 3D point clouds; and performs spatial downsampling fusion on the multiple second 3D point clouds based on hash mapping to obtain a fused 3D point cloud. This method does not require a complex neural network model. Back-projection based on the pinhole camera's geometric model allows all RGB pixels to be mapped to 3D points using only linear arithmetic operations, reducing computational complexity. Furthermore, spatial downsampling fusion of the multiple second 3D point clouds based on hash mapping further reduces computational complexity while ensuring point cloud accuracy, thus achieving real-time point cloud reconstruction.

[0042] Example 2 If, in step 101 above, a depth map corresponding to the RGB image is acquired by a depth camera (i.e., a depth map corresponding to the RGB image is acquired through a depth camera), then before step 102 above, such as Figure 2 As shown, it may also include pixel alignment of the depth map in the following manner: Step 201: Using the pre-acquired depth camera intrinsic parameter information, back-project the depth pixels in the depth map corresponding to the RGB image to the depth camera coordinate system to obtain the three-dimensional depth coordinates in the depth camera coordinate system. Step 202: Using the coordinate transformation parameters between the pre-acquired RGB image and depth map, as well as the RGB camera intrinsic information, the three-dimensional depth coordinates are transformed and projected onto the RGB image plane to obtain the RGB pixel index corresponding to the depth pixel; Step 203: When multiple depth pixels correspond to the same RGB pixel index, retain the depth pixel with the smallest depth value to obtain a depth map aligned with the RGB image pixels. Step 102 above may specifically include: Based on the geometric model of the pinhole camera, pixel back projection is performed on the RGB image and the depth map aligned with the pixels of the RGB image to obtain the first three-dimensional point cloud of each RGB camera.

[0043] In step 201, the depth pixels in the depth map corresponding to the RGB image can be back-projected to the depth camera coordinate system using the pre-acquired depth camera intrinsic parameter information to obtain the three-dimensional depth coordinates in the depth camera coordinate system.

[0044] In practical implementation, the depth pixels in the depth map are ( , The corresponding depth value is z. d .

[0045] Depth camera intrinsics include: For the depth camera focal length, Let be the optical center coordinates of the depth camera. The depth pixels in the depth map are back-projected to the depth camera coordinate system using the following formula: Wherein, the three-dimensional depth coordinates in the depth camera coordinate system are = ( , , ).

[0046] In step 202, the three-dimensional depth coordinates are transformed and projected onto the RGB image plane using the coordinate transformation parameters between the pre-acquired RGB image and depth map, as well as the RGB camera intrinsic information, to obtain the RGB pixel index corresponding to the depth pixel.

[0047] In practice, the coordinate transformation parameters can include the rotation matrix R{d→c} from the depth camera to the RGB camera and the translation vector t{d→c}, which can be obtained through factory calibration or pre-calibrated in other ways.

[0048] First, according to the following formula, each three-dimensional depth coordinate is... Transform to the RGB camera coordinate system to obtain the three-dimensional coordinates of the RGB camera coordinate system. = : Then, using the RGB camera intrinsic information ( , , The three-dimensional coordinates are expressed by the following formula. Projecting onto the RGB image plane yields two-dimensional coordinates. , ): Finally, for , Rounding down gives the depth pixels. , The corresponding RGB pixel index.

[0049] In step 203, when multiple depth pixels correspond to the same RGB pixel index, the depth pixel with the smallest depth value is retained to obtain a depth map aligned with the RGB image pixels.

[0050] In practice, due to the difference in viewing angle between the depth camera and the RGB camera, multiple depth pixels may be projected onto the same position on the RGB image plane. This embodiment uses a scatter-reduce-min operation to take the minimum value of all depth values ​​under the same RGB pixel index, that is, to retain the depth value closest to the camera.

[0051] In this way, pixel alignment of the depth map is performed using the above method, avoiding pixel offset between the depth map and the RGB image due to differences in the optical center position or resolution between the depth camera and the RGB camera, thereby further improving the accuracy of point cloud reconstruction.

[0052] Example 3 After step 102 above, as Figure 3 As shown, it also includes correcting the RGB values ​​in the first 3D point cloud in the following manner: Step 301: Based on the depth value of each three-dimensional point in the first three-dimensional point cloud, the first three-dimensional point cloud is layered to obtain multi-layer sub-three-dimensional point clouds; Step 302: Calculate the brightness of each three-dimensional point in each sub-three-dimensional point cloud, and calculate the median and absolute median difference of brightness of each sub-three-dimensional point cloud based on the brightness of each three-dimensional point. Step 303: Determine the brightness scaling factor of each sub-3D point cloud based on the median brightness and absolute median difference of each sub-3D point cloud layer. Step 304: Scale the RGB values ​​of each sub-3D point cloud using the brightness scaling factor of each sub-3D point cloud to obtain the first 3D point cloud after brightness correction.

[0053] In step 301, the first three-dimensional point cloud is layered according to the depth value of each three-dimensional point in the first three-dimensional point cloud to obtain a multi-layered sub-three-dimensional point cloud.

[0054] In practice, the first 3D point cloud can be divided into multiple layers based on a preset depth range and the depth values ​​(i.e., the Zc components of the 3D points). For example, the 3D points in the first 3D point cloud can be divided into five sub-3D point clouds according to Zc: [0, 0.5m), [0.5m, 1m), [1m, 2m), [2m, 4m), and [4m, +∞). Layer 0: Zc∈[0,0.5m); Layer 1: Zc∈[0.5,1.0m); Layer 2: Zc∈[1.0,2.0m); Layer 3: Zc∈[2.0,4.0m); Layer 4: Zc∈[4.0m,+∞).

[0055] In step 302, the brightness of each three-dimensional point in each sub-three-dimensional point cloud is calculated, and the median and absolute median difference of the brightness of each sub-three-dimensional point cloud are calculated based on the brightness of each three-dimensional point.

[0056] In practice, the brightness value L of each three-dimensional point can be calculated using the following brightness formula: L = 0.299 × R + 0.587 × G + 0.114 × B Among them, R, G, and B can be the color component values ​​of a three-dimensional point, which can be obtained directly when the RGB image is acquired.

[0057] The median brightness of each sub-3D point cloud layer can be calculated using the following median formula. k : The set of all brightness values ​​of the k-th layer {L i}: median k =median({L i}) The absolute median (MAD) of each sub-3D point cloud layer can be calculated using the following formula. k ): MAD k =median({|L i -median k |}) In step 303 above, the brightness scaling factor of each sub-3D point cloud is determined based on the median brightness and absolute median difference of each sub-3D point cloud layer.

[0058] In practice, the target brightness L of each sub-3D point cloud layer is first determined. t Target brightness L tIt can be preset or set based on the average brightness of the 3D point cloud of that layer. There is no limitation on the target brightness setting here; it can be set based on actual needs.

[0059] Then, the brightness scaling factor is calculated using the following formula. : In this embodiment of the invention, an absolute median difference threshold may also be set. If MAD k Greater than or equal to This indicates that the statistics are reliable, and the brightness scaling factor is [value missing]. Valid; if MAD k Less than This indicates that the statistics are unreliable, and the brightness scaling factor is... Invalid. Use the global default brightness scaling factor (usually 1).

[0060] In step 304 above, the RGB values ​​of each sub-3D point cloud are scaled using the brightness scaling factor of each sub-3D point cloud to obtain the first 3D point cloud after brightness correction.

[0061] In practice, methods such as Gamma specular compression and Alpha original image blending can be used to scale the RGB values ​​of the 3D points in each sub-3D point cloud layer based on the brightness scaling factor. After correction, the RGB values ​​are constrained to the standard range of 0 to 1. The corrected RGB values ​​are then incorporated into the subsequent registration and fusion process along with the first 3D point cloud. The 3D coordinates are unaffected by color correction.

[0062] In one embodiment, after step 303 above, the method further includes: Based on a preset scaling range, the brightness scaling factor of each sub-3D point cloud is limited so that the brightness scaling factor of each sub-3D point cloud is within the preset scaling range.

[0063] In practice, to ensure global brightness balance, the calculated brightness scaling factor can be limited based on a preset scaling range, such as [0.6, 1.6], to prevent image distortion. Specifically, if the brightness scaling factor is greater than 1.6, it is limited to 1.6; if the brightness scaling factor is less than 0.6, it is limited to 0.6.

[0064] In this way, by correcting the RGB values ​​of the first 3D point cloud in the above manner, the differences in brightness and exposure caused by RGB camera acquisition can be eliminated. Moreover, the median and absolute median difference are robust to outliers. Even if there are a few highlights or overexposed points, they will not significantly affect the statistical results, further improving the accuracy of point cloud reconstruction.

[0065] Example 4 Step 104 above, as follows Figure 4 As shown, it can specifically include: Step 401: Merge multiple second three-dimensional point clouds, and divide the merged second three-dimensional point clouds into at least one point cloud set according to a preset depth segmentation threshold. Step 402: For each point cloud set, discretize the coordinates of each 3D point in the set according to the preset voxel size corresponding to the point cloud set to determine the voxel identifier; determine the hash index according to the voxel identifier of each 3D point; map 3D points with the same hash index to the same hash bucket; filter multiple 3D points in each hash bucket to determine at least one target 3D point from each hash bucket; wherein the hash bucket intervals of different point cloud sets are isolated from each other. Step 403: The target 3D points of each hash bucket are stitched together to obtain the fused 3D point cloud.

[0066] In step 401 above, multiple second three-dimensional point clouds are merged, and the merged second three-dimensional point clouds are divided into at least one point cloud set according to a preset depth segmentation threshold.

[0067] In practice, multiple second-dimensional point clouds can be merged into a unified point cloud set first. Then, according to a preset depth segmentation threshold, the unified point cloud set (i.e., the merged second-dimensional point cloud) is divided into at least one point cloud set. Specifically, this can be done based on the Z-values ​​of the 3D points in the second-dimensional point cloud.

[0068] For example, in this embodiment, the depth segmentation threshold is set to 0.5m, dividing the merged second 3D point cloud into at least one point cloud set: Point cloud set 1: A set of 3D points with depth values ​​in the range [0, 0.5m), with a voxel size of 5mm; Point cloud set 2: A set of 3D points with depth values ​​in the range of [0.5m, 2.0m), and voxel size of 10mm, etc.

[0069] In step 402 above, for each point cloud set, the coordinates of each three-dimensional point in the set are discretized according to the preset voxel size corresponding to the point cloud set to determine the voxel identifier.

[0070] In practice, for each 3D point in a point cloud set, the coordinates of the 3D points are rounded down according to the voxel size corresponding to the set to obtain the voxel identifier: v x =floor(X / voxel size ) v y =floor(Y / voxel size ) v z =floor(Z / voxel size ) Where floor() represents the floor function, voxel size The preset voxel size is represented by (X, Y, Z), which represents the coordinates of a 3D point in the point cloud set. x v y v z ) is the voxel identifier corresponding to (X, Y, Z).

[0071] Then, the hash index is determined based on the voxel identifier of each 3D point, which may specifically include: Using preset weights, the voxel identifiers of each 3D point are weighted, and then the weighted voxel identifiers are processed using an XOR operation to obtain the hash value of each 3D point. Based on the hash value of each 3D point and the preset hash bucket capacity, the bucket offset of the 3D point is determined. The bucket offset of the 3D point is added to the starting offset of the corresponding gear in the point cloud set to obtain the hash index of the 3D point.

[0072] In practice, a preset weight can be used to weight the voxel identifier of each 3D point, and then an XOR operation can be used to process the weighted voxel identifier to obtain the hash value of each 3D point. In order to break the spatial regularity of 3D coordinates, adjacent / simple arithmetic progression voxel coordinates are scattered to make the hash values ​​as evenly distributed as possible and reduce clustering conflicts. The preset weight can be a large prime number. For example, the weights of each component of the voxel identifier can be set to P1=73856093, P2=19349663, and P3=83492791.

[0073] Each component of the voxel label is multiplied by a different large prime number weight: h1=v x ×P1, h2=v y ×P2,h3=v z ×P3 Then, by performing an XOR merge operation, the hash value of the 3D point can be obtained: hash=h1⊕h2⊕h3 Where ⊕ represents a bitwise XOR operation, and hash is the hash value.

[0074] Next, the offset within the bucket is obtained by taking the modulus: offset = hash & (capacity - 1) Where capacity is the hash bucket capacity for each tier, and the hash bucket capacity can be set to an integer power of 2 (such as 1024, 65536, etc.). offset is the bucket offset of the 3D point.

[0075] Finally, the bucket offset of the 3D point is added to the starting offset of the gear corresponding to the point cloud set to obtain the hash index global_index of the 3D point.

[0076] global_index=base_offset+offset Where base_offset is the starting offset of the gear corresponding to this point cloud set.

[0077] In practice, hash bucket capacities can be allocated to different point cloud sets. For example, if the total hash bucket capacity is 2048, the hash bucket capacities for point cloud set 1 and point cloud set 2 can be the same or different. The hash bucket capacity for point cloud set 1 is 1024, the bucket range is [0, 1023], and the starting offset for the depth level is 0. The hash bucket capacity for point cloud set 2 is 1024, the bucket range is [1024, 2047], and the starting offset for the depth level is 1024. In this way, the hash bucket ranges of different point cloud sets are isolated from each other and do not overlap, which can ensure that 3D points at different depth levels will not fall into the same hash bucket and avoid erroneous deduplication caused by different voxel sizes.

[0078] After obtaining the voxel identifier of each 3D point and determining its hash index, 3D points with the same hash index can be mapped to the same hash bucket. For multiple 3D points within the same hash bucket, a minimum reduction operation can be used to retain the 3D point with the smallest depth value as the target 3D point. This operation is executed in parallel atomically using the GPU's scatter_reduce operator, without requiring a global sort of the 3D points within the bucket, resulting in linear deduplication complexity.

[0079] Finally, in step 403 above, the target 3D points of each hash bucket are stitched together to obtain the fused 3D point cloud.

[0080] In this way, by performing spatial downsampling and fusion of multiple second-dimensional point clouds based on hash mapping as described above, a fused 3D point cloud can be obtained, which can remove redundant points in the overlapping areas of multiple views and reduce the amount of data in the 3D point cloud.

[0081] Example 5 Step 104 above, as follows Figure 5 As shown, it can specifically include: Step 501: Merge multiple second three-dimensional point clouds, and divide the merged second three-dimensional point clouds into a first point cloud set and at least one second point cloud set according to a preset depth segmentation threshold. Step 502: For each second point cloud set, discretize the coordinates of each 3D point in the set according to the preset voxel size corresponding to the second point cloud set to determine the voxel identifier; determine the hash index according to the voxel identifier of each 3D point; map 3D points with the same hash index to the same hash bucket; filter multiple 3D points in each hash bucket to determine at least one target 3D point from each hash bucket; wherein the hash bucket intervals of different second point cloud sets are isolated from each other; Step 503: The target 3D points of each hash bucket and the 3D points of the first point cloud set are stitched together to obtain the fused 3D point cloud.

[0082] Unlike Implementation Four, in step 501 above, the merged second 3D point cloud is divided into a first point cloud set, i.e., a near-field point cloud set, and at least one second point cloud set, i.e., a far-field point cloud set. For the near-field point cloud set, all 3D points in the near-field point cloud set are retained. For each far-field point cloud set, downsampling is performed using the same method as in Implementation Four. The specific implementation process can be referred to in Implementation Four above, and will not be elaborated further here.

[0083] In this way, preserving the near-view point cloud ensures detailed information about nearby objects and avoids accuracy loss due to voxelization, while downsampling the far-view point cloud removes redundant points in overlapping areas of multiple views, reducing the amount of data. This, in turn, improves the accuracy and efficiency of point cloud reconstruction.

[0084] Example 6 This invention also provides a three-dimensional point cloud reconstruction device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to the three-dimensional point cloud reconstruction method described above, the implementation of this method can be found in the implementation of the three-dimensional point cloud reconstruction method, and repeated details will not be elaborated further.

[0085] like Figure 6 The diagram shown is a schematic representation of a three-dimensional point cloud reconstruction device provided in an embodiment of the present invention. The device may include: Image acquisition module 601 is used to acquire RGB images from multiple RGB cameras and depth maps corresponding to the RGB images; The back projection module 602 is used to perform pixel back projection on the RGB image and depth map based on the geometric model of the pinhole camera to obtain the first three-dimensional point cloud of each RGB camera. The registration module 603 is used to register multiple first three-dimensional point clouds based on multiple pre-acquired RGB camera extrinsic information to obtain multiple registered second three-dimensional point clouds. The fusion module 604 is used to perform spatial downsampling fusion of multiple second 3D point clouds based on hash mapping to obtain a fused 3D point cloud.

[0086] In one embodiment, the image acquisition module 601 may specifically include: Obtain the depth map corresponding to the RGB image captured by the depth camera; The aforementioned 3D point cloud reconstruction device also includes a depth alignment module, used before the back projection module 602 performs pixel back projection on the RGB image and depth map based on the pinhole camera geometric model to obtain the first 3D point cloud of each RGB camera: Using the pre-acquired intrinsic parameters of the depth camera, the depth pixels in the depth map corresponding to the RGB image are back-projected to the depth camera coordinate system to obtain the three-dimensional depth coordinates in the depth camera coordinate system. Using the coordinate transformation parameters between the pre-acquired RGB image and depth map, as well as the RGB camera intrinsic parameters, the three-dimensional depth coordinates are transformed and projected onto the RGB image plane to obtain the RGB pixel index corresponding to the depth pixel; When multiple depth pixels correspond to the same RGB pixel index, retain the depth pixel with the smallest depth value to obtain a depth map aligned with the RGB image pixels; The back projection module 602 can be specifically used to: perform pixel back projection on the RGB image and the depth map aligned with the pixels of the RGB image based on the geometric model of the pinhole camera, so as to obtain the first three-dimensional point cloud of each RGB camera.

[0087] In one embodiment, after the back-projection module 602 performs pixel back-projection on the RGB image and depth map based on the pinhole camera geometric model to obtain a first three-dimensional point cloud for each RGB camera, the following steps are further performed for each first three-dimensional point cloud: Based on the depth value of each three-dimensional point in the first three-dimensional point cloud, the first three-dimensional point cloud is layered to obtain multi-layered sub-three-dimensional point clouds. Calculate the brightness of each 3D point in each sub-3D point cloud layer, and calculate the median and absolute median difference of brightness of each sub-3D point cloud layer based on the brightness of each 3D point. The brightness scaling factor for each sub-3D point cloud is determined based on the median brightness and absolute median difference of each sub-3D point cloud layer. By using the brightness scaling factor of each sub-3D point cloud, the RGB values ​​of each sub-3D point cloud are scaled to obtain the first 3D point cloud after brightness correction.

[0088] In one embodiment, after determining the brightness scaling factor of each sub-3D point cloud based on the brightness median and absolute median difference of each sub-3D point cloud layer, the method further includes: Based on a preset scaling range, the brightness scaling factor of each sub-3D point cloud is limited so that the brightness scaling factor of each sub-3D point cloud is within the preset scaling range.

[0089] In one embodiment, the fusion module 604 can be specifically used for: Multiple second-dimensional point clouds are merged, and the merged second-dimensional point clouds are divided into at least one point cloud set according to a preset depth segmentation threshold. For each point cloud set, the coordinates of each 3D point in the set are discretized according to the preset voxel size corresponding to the point cloud set to determine the voxel identifier; the hash index is determined according to the voxel identifier of each 3D point; 3D points with the same hash index are mapped to the same hash bucket; multiple 3D points in each hash bucket are filtered to determine at least one target 3D point from each hash bucket; wherein the hash bucket intervals of different point cloud sets are isolated from each other. The target 3D points of each hash bucket are stitched together to obtain a fused 3D point cloud.

[0090] In one embodiment, the fusion module 604 can be specifically used for: Multiple second-dimensional point clouds are merged, and the merged second-dimensional point clouds are divided into a first point cloud set and at least one second point cloud set according to a preset depth segmentation threshold. For each second point cloud set, the coordinates of each 3D point in the set are discretized according to the preset voxel size corresponding to the second point cloud set to determine the voxel identifier; a hash index is determined according to the voxel identifier of each 3D point; 3D points with the same hash index are mapped to the same hash bucket; multiple 3D points in each hash bucket are filtered to determine at least one target 3D point from each hash bucket; wherein the hash bucket intervals of different second point cloud sets are isolated from each other; The target 3D points of each hash bucket and the 3D points of the first point cloud set are stitched together to obtain the fused 3D point cloud.

[0091] In one embodiment, the fusion module 604 determines the hash index based on the voxel identifier of each 3D point, including: Using preset weights, the voxel identifiers of each 3D point are weighted, and then the weighted voxel identifiers are processed using an XOR operation to obtain the hash value of each 3D point. Based on the hash value of each 3D point and the preset hash bucket capacity, the bucket offset of the 3D point is determined. The bucket offset of the 3D point is added to the starting offset of the corresponding gear in the point cloud set to obtain the hash index of the 3D point.

[0092] Example 7 This invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described three-dimensional point cloud reconstruction method.

[0093] Example 8 This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described three-dimensional point cloud reconstruction method.

[0094] Example 9 This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described three-dimensional point cloud reconstruction method.

[0095] In this embodiment of the invention, RGB images from multiple RGB cameras and corresponding depth maps are acquired. Based on the geometric model of the pinhole camera, pixel back-projection is performed on the RGB images and depth maps to obtain a first 3D point cloud for each RGB camera. Based on pre-acquired extrinsic information from multiple RGB cameras, the multiple first 3D point clouds are registered to obtain multiple registered second 3D point clouds. Spatial downsampling and fusion of the multiple second 3D point clouds are performed based on hash mapping to obtain a fused 3D point cloud. This method does not require a complex neural network model. Back-projection based on the geometric model of the pinhole camera allows all RGB pixels to be mapped to 3D points using only linear arithmetic operations, reducing computational complexity. Furthermore, spatial downsampling and fusion of the multiple second 3D point clouds based on hash mapping can further reduce computational complexity while ensuring point cloud accuracy, thereby achieving real-time point cloud reconstruction.

[0096] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for reconstructing three-dimensional point clouds, characterized in that, include: Acquire RGB images from multiple RGB cameras and depth maps corresponding to the RGB images; Based on the geometric model of the pinhole camera, pixel back projection is performed on the RGB image and depth map to obtain the first three-dimensional point cloud of each RGB camera; Based on the pre-acquired extrinsic information of multiple RGB cameras, multiple first 3D point clouds are registered to obtain multiple registered second 3D point clouds. Multiple second-dimensional point clouds are spatially downsampled and fused based on hash mapping to obtain a fused 3D point cloud.

2. The method according to claim 1, characterized in that, Obtaining the depth map corresponding to the RGB image includes: Obtain the depth map corresponding to the RGB image captured by the depth camera; Before performing pixel back projection on the RGB image and depth map based on the pinhole camera geometric model to obtain the first 3D point cloud of each RGB camera, the process also includes: Using the pre-acquired intrinsic parameters of the depth camera, the depth pixels in the depth map corresponding to the RGB image are back-projected to the depth camera coordinate system to obtain the three-dimensional depth coordinates in the depth camera coordinate system. Using the coordinate transformation parameters between the pre-acquired RGB image and depth map, as well as the RGB camera intrinsic parameters, the three-dimensional depth coordinates are transformed and projected onto the RGB image plane to obtain the RGB pixel index corresponding to the depth pixel; When multiple depth pixels correspond to the same RGB pixel index, retain the depth pixel with the smallest depth value to obtain a depth map aligned with the RGB image pixels; Based on the geometric model of the pinhole camera, pixel back projection is performed on the RGB image and depth map to obtain the first 3D point cloud of each RGB camera, including: Based on the geometric model of the pinhole camera, pixel back projection is performed on the RGB image and the depth map aligned with the pixels of the RGB image to obtain the first three-dimensional point cloud of each RGB camera.

3. The method as described in claim 1, characterized in that, Based on the geometric model of the pinhole camera, after performing pixel back projection on the RGB image and depth map to obtain the first 3D point cloud of each RGB camera, the following steps are also performed on each first 3D point cloud: Based on the depth value of each three-dimensional point in the first three-dimensional point cloud, the first three-dimensional point cloud is layered to obtain multi-layered sub-three-dimensional point clouds. Calculate the brightness of each 3D point in each sub-3D point cloud layer, and calculate the median and absolute median difference of brightness of each sub-3D point cloud layer based on the brightness of each 3D point. The brightness scaling factor for each sub-3D point cloud is determined based on the median brightness and absolute median difference of each sub-3D point cloud layer. By using the brightness scaling factor of each sub-3D point cloud, the RGB values ​​of each sub-3D point cloud are scaled to obtain the first 3D point cloud after brightness correction.

4. The method as described in claim 3, characterized in that, After determining the brightness scaling factor for each sub-3D point cloud layer based on the median brightness and absolute median difference of each layer, the following steps are also included: Based on a preset scaling range, the brightness scaling factor of each sub-3D point cloud is limited so that the brightness scaling factor of each sub-3D point cloud is within the preset scaling range.

5. The method as described in claim 1, characterized in that, Based on hash mapping, multiple second-dimensional point clouds are spatially downsampled and fused to obtain a fused 3D point cloud, including: Multiple second-dimensional point clouds are merged, and the merged second-dimensional point clouds are divided into at least one point cloud set according to a preset depth segmentation threshold. For each point cloud set, the coordinates of each 3D point in the set are discretized according to the preset voxel size corresponding to the point cloud set to determine the voxel identifier; the hash index is determined according to the voxel identifier of each 3D point; 3D points with the same hash index are mapped to the same hash bucket; multiple 3D points in each hash bucket are filtered to determine at least one target 3D point from each hash bucket; wherein the hash bucket intervals of different point cloud sets are isolated from each other. The target 3D points of each hash bucket are stitched together to obtain a fused 3D point cloud.

6. The method as described in claim 1, characterized in that, Based on hash mapping, multiple second-dimensional point clouds are spatially downsampled and fused to obtain a fused 3D point cloud, including: Multiple second-dimensional point clouds are merged, and the merged second-dimensional point clouds are divided into a first point cloud set and at least one second point cloud set according to a preset depth segmentation threshold. For each second point cloud set, the coordinates of each 3D point in the set are discretized according to the preset voxel size corresponding to the second point cloud set to determine the voxel identifier; a hash index is determined according to the voxel identifier of each 3D point; 3D points with the same hash index are mapped to the same hash bucket; multiple 3D points in each hash bucket are filtered to determine at least one target 3D point from each hash bucket; wherein the hash bucket intervals of different second point cloud sets are isolated from each other; The target 3D points of each hash bucket and the 3D points of the first point cloud set are stitched together to obtain the fused 3D point cloud.

7. The method as described in claim 5 or 6, characterized in that, The hash index is determined based on the voxel identifier of each 3D point, including: Using preset weights, the voxel identifiers of each 3D point are weighted, and then the weighted voxel identifiers are processed using an XOR operation to obtain the hash value of each 3D point. Based on the hash value of each 3D point and the preset hash bucket capacity, the bucket offset of the 3D point is determined. The bucket offset of the 3D point is added to the starting offset of the corresponding gear in the point cloud set to obtain the hash index of the 3D point.

8. A three-dimensional point cloud reconstruction device, characterized in that, include: An image acquisition module is used to acquire RGB images from multiple RGB cameras and depth maps corresponding to the RGB images; The back projection module is used to perform pixel back projection on the RGB image and depth map based on the geometric model of the pinhole camera to obtain the first three-dimensional point cloud of each RGB camera. The registration module is used to register multiple first 3D point clouds based on multiple pre-acquired RGB camera extrinsic information to obtain multiple registered second 3D point clouds. The fusion module is used to perform spatial downsampling fusion of multiple second-dimensional point clouds based on hash mapping to obtain a fused 3D point cloud.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the three-dimensional point cloud reconstruction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the three-dimensional point cloud reconstruction method according to any one of claims 1 to 7.

Images