A fusion perception model training method and device based on visual images and point clouds, a dense three-dimensional point cloud acquisition method and device, and a modeling method and device

Abstract

The application discloses a fusion perception model training method and device based on visual images and point clouds, and comprises the following steps: a Transformer model is built, the Transformer model comprises an encoder, a decoder and an up-sampling layer connected in sequence, the encoder comprises a certain number of feature extraction layers, the feature extraction layer comprises a multi-head self-attention module and a multi-layer perception module connected in sequence, the multi-head self-attention module and the multi-layer perception module are both connected with a LayerNorm layer in front and a residual connection in back; the decoder comprises a certain number of decoding layers, the decoding layer comprises a two-dimensional convolution module and an up-sampling module connected in sequence; the output of the decoder is recovered to the size of the original image through the up-sampling layer of the bilinear interpolation, and the final dense three-dimensional point cloud is obtained. According to the method and device, the visual image and the point cloud image are respectively converted into corresponding one-dimensional sequences, samples are generated according to the one-dimensional sequences, the workload of sample marking is avoided compared with the prior art which needs manual marking of samples, and then the training samples can be generated more efficiently, and the training efficiency of the model is improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a method and apparatus for training a fusion perception model based on visual images and point clouds, acquiring dense 3D point clouds, and modeling. Background Technology

[0002] LiDAR and cameras are the most commonly used sensors in computer vision. LiDAR can quickly and accurately acquire 3D environmental information, but the resulting 3D point clouds are relatively sparse and only contain depth information. Cameras, on the other hand, can acquire dense and rich information, but struggle to capture the depth information of objects within their field of view. Fusion of these two technologies yields more ideal 3D reconstruction results; this process is known as sensor fusion technology. Utilizing multiple sensors with complementary characteristics to enhance perception capabilities and reduce costs has become an emerging research topic. The complementary nature of cameras and LiDAR makes fusion models more effective and popular than other sensor fusion configurations.

[0003] The prior art patent application CN202010058810.8 discloses a learning method and learning device, as well as a testing method and testing device using the same. It integrates an image generated by a camera and a corresponding point cloud image generated by radar or lidar, according to their respective convolutional steps of a neural network. The process includes: (a) a computing device using at least one initial operational layer to integrate at least one original image generated by the camera with at least one original point cloud image generated by the radar or lidar, (i) adding depth information contained in the original point cloud image to the original image to generate at least one first fused feature map, and (ii) adding color information contained in the original image to the original point cloud image to generate at least one first fused point cloud image. This method uses a CNN convolutional neural network model for depth information fusion. The process involves first labeling the depth information in a two-dimensional image, then training the CNN model using the two-dimensional image, and then using the CNN model to fuse the image and point cloud. Its drawback is that the CNN training process requires manual labeling of depth information, resulting in low training efficiency. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of low fusion efficiency in the existing technology, and to provide a method and device for training a fusion perception model based on visual images and point clouds, acquiring dense 3D point clouds, and modeling, which can improve the training efficiency of the model.

[0005] The present invention adopts the following technical solution:

[0006] An improved method for training a fusion perception model based on visual images and point clouds includes the following steps:

[0007] A Transformer model is constructed, comprising an encoder, a decoder, and an upsampling layer connected in series. The encoder includes a number of feature extraction layers, each containing a multi-head self-attention module and a multi-layer perceptron module connected in series. Both the multi-head self-attention module and the multi-layer perceptron module are connected to a LayerNorm layer before and a residual connection after. The decoder includes a number of decoding layers, each containing a cascaded 2D convolutional module and an upsampling module. The output of the decoder is then upsampled using bilinear interpolation to restore the original image size, resulting in the final dense 3D point cloud.

[0008] Visual images and point cloud images are converted into corresponding one-dimensional sequences. Samples are generated based on the one-dimensional sequences. The Transformer model is trained using the sample set composed of these samples until the model converges, resulting in a fusion perception model.

[0009] Furthermore, the multilayer perceptron module consists of two nonlinear layers connected in series with GELU.

[0010] Furthermore, a BatchNorm layer is provided between the two-dimensional convolution module and the upsampling module.

[0011] Furthermore, the visual image and the point cloud image are data with the same shooting parameters, which include: shooting direction and shooting coordinates.

[0012] Furthermore, the step of generating samples based on a one-dimensional sequence includes:

[0013] One-dimensional sequences are concatenated to obtain two-dimensional concatenation results. The two-dimensional concatenation results are then trimmed into blocks, and finally each block is mapped to a one-dimensional space.

[0014] The blocks mapped to one-dimensional space are mapped to D-dimensional block embedding information, and positional encoding is added to each block embedding information to obtain samples.

[0015] An improved method for acquiring dense 3D point clouds based on visual images and point clouds is as follows: after acquiring the visual image and point cloud image to be fused, the image is converted into data to be processed and input into the fusion perception model obtained above to obtain the depth information of each pixel in the visual image to be fused. Based on the depth information of each pixel, each pixel is mapped to the point cloud space to obtain a dense 3D point cloud.

[0016] An improved modeling method based on visual images and point clouds is as follows: based on the depth information of each point in the dense 3D point cloud obtained above, the plane in which each point is located is extracted, and each plane is subjected to 2D expansion processing to obtain candidate planes; based on the geometric relationship between each candidate plane, a spatial model closed by the candidate planes is obtained.

[0017] A training device for a fusion perception model based on visual images and point clouds, used in the above training method, wherein the improvement is that the device includes:

[0018] The model building module is used to build a Transformer model, which includes an encoder, a decoder, and an upsampling layer connected in series. The encoder includes a number of feature extraction layers, which contain a series of multi-head self-attention modules and multi-layer perceptron modules. Both the multi-head self-attention modules and multi-layer perceptron modules are connected to a LayerNorm layer before and a residual connection after. The decoder includes a number of decoding layers, which contain a series of two-dimensional convolutional modules and an upsampling module. The output of the decoder is then upsampled by a bilinear interpolation layer to restore the original image size, resulting in the final dense three-dimensional point cloud.

[0019] The sample generation and model training module is used to convert visual images and point cloud images into corresponding one-dimensional sequences, generate samples based on the one-dimensional sequences, and train the built Transformer model using the sample set composed of samples until the model converges, thus obtaining the fused perception model.

[0020] An improved device for acquiring dense 3D point clouds based on visual images and point clouds, wherein the device comprises:

[0021] The acquisition module is used to acquire the visual image and point cloud image to be fused, convert them into data to be processed, and input them into the fusion perception model obtained above to obtain the depth information of each pixel in the visual image to be fused.

[0022] The mapping module is used to map each pixel into the point cloud space based on the depth information of each pixel, so as to obtain a dense three-dimensional point cloud.

[0023] An improved modeling apparatus based on visual images and point clouds, wherein the apparatus comprises:

[0024] The extraction module is used to extract the plane where each point is located based on the depth information of each point in the dense 3D point cloud obtained above.

[0025] The expansion module is used to perform two-dimensional expansion processing on each plane to obtain candidate planes;

[0026] The modeling module is used to obtain a spatial model enclosed by the candidate planes based on the geometric relationships between them.

[0027] The beneficial effects of this invention are:

[0028] The method and apparatus disclosed in this invention convert visual images and point cloud images into corresponding one-dimensional sequences, and generate samples based on the one-dimensional sequences. Compared with the prior art, which requires manual labeling of samples, this method avoids the workload of sample labeling, thereby generating training samples more efficiently and improving the training efficiency of the model. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the training method disclosed in this invention;

[0030] Figure 2 This is a schematic diagram of the Transformer model structure in the training method disclosed in this invention;

[0031] Figure 3 This is a schematic diagram of a point cloud image used to generate samples in the training method disclosed in this invention;

[0032] Figure 4 This is a schematic diagram illustrating the principle of the modeling method disclosed in this invention;

[0033] Figure 5 This is a schematic diagram of the results of the modeling method disclosed in this invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0035] Example 1 discloses a method for training a fusion perception model based on visual images and point clouds, such as... Figure 1 As shown, it includes the following steps:

[0036] Step 1, build the Transformer model (a novel network architecture for depth estimation), such as... Figure 2As shown, the Transformer model includes an encoder 20, a decoder 21, and an upsampling layer 23 connected in series. The encoder includes six feature extraction layers, each containing a multi-head self-attention module 2003 and a multilayer perceptron module 2007 connected in series. The multi-head self-attention module is pre-connected to the first LayerNorm layer 2001, and the multilayer perceptron module is pre-connected to the second LayerNorm layer 2005. Both the multi-head self-attention module and the multilayer perceptron module are post-connected to residual connections (residual connection is a technical term).

[0037] The multilayer perceptron module contains two nonlinear layers with GELUs, namely:

[0038] Multi-head self-attention module: z′ l =MSA(LN(z) l-1 ))+z l-1 l = 1, 2, 3...6

[0039] Multilayer perceptron module: z l =MLP(LN(z′) l ))+z′ l l = 1, 2, 3...6

[0040] The final output of the encoder is denoted as y, where y = LN(z6).

[0041] The decoder includes three decoding layers, each consisting of a cascaded two-dimensional convolutional module 2101 and an upsampling module 2103.

[0042] A BatchNorm (BN) layer is placed between the 2D convolution module and the upsampling module to prevent overfitting.

[0043] The decoder output is upsampled by bilinear interpolation layer 2105 to restore the original image size, resulting in the final dense 3D point cloud (depth map);

[0044] Step 2: Convert the visual image and point cloud image into corresponding one-dimensional sequences. Generate samples based on the one-dimensional sequences, and train the constructed Transformer model using the sample set composed of these samples until the model converges, thus obtaining the fused perception model. The visual image and point cloud image are data with the same shooting parameters, including shooting direction and shooting coordinates.

[0045] Point cloud images used to generate samples, such as Figure 3 As shown, the point cloud image has a sparse density and uneven distribution, making it unsuitable for modeling. Therefore, the point cloud image needs to be processed.

[0046] First, convert the high-precision RGB image X1∈R H×W×3And sparse depth map (point cloud image) X2∈R H×W×1 Together they serve as input X to the Transformer model, X = [X1; X2] ∈ R. H×W×4 Where H is the height of the visual image and W is the width of the visual image.

[0047] Since the standard Transformer model accepts a sequence of 1-dimensional token embeddings as input, in order to process 2-dimensional input X∈R H×W×4 Cut it into 3x3 patches, and perform a transformation operation on each patch to flatten it into a one-dimensional sequence, i.e.:

[0048]

[0049] Then, each patch is linearly projected to map it to D-dimensional patch embeddingsX. e ,Right now:

[0050]

[0051] And add a learnable positional code E to each encoded patch. pos E pos ∈R 9×D This is used to preserve location information, thereby obtaining a sample.

[0052] Several visual images and corresponding point cloud images can each generate several samples, which together form a sample set.

[0053] The sample set is input into the encoder, which is based on the Transformer, for encoding. This initial input is denoted as z0.

[0054] z0 = X e +E pos , z0∈R 9×D

[0055] The fusion perception model trained in this embodiment uses high-resolution images to guide 3D depth upsampling, aiming to upsample sparse, irregular depths to dense, regular depths, which can reduce the severe uneven distribution in point clouds acquired by LiDAR scanning.

[0056] Example 2 discloses a method for acquiring dense 3D point clouds based on visual images and point clouds: After acquiring the visual image and point cloud image to be fused, they are converted into data to be processed and input into the fusion perception model obtained in Example 1 to obtain the depth information of each pixel in the visual image to be fused. Based on the depth information of each pixel, each pixel is mapped to the point cloud space to obtain a dense 3D point cloud.

[0057] The real-world environment is recorded by projecting it onto an irregular, disordered, continuous image plane, while simultaneously utilizing regular, ordered, discrete sparse point clouds that preserve 3D geometry. These differences in features between point cloud images and visual images lead to different feature extraction methods. Therefore, this embodiment fuses two different types of data to obtain samples, thereby training a model capable of obtaining dense 3D point clouds using both sparse point cloud images and visual images.

[0058] Example 3 discloses a modeling method based on visual images and point clouds: Based on the dense 3D point cloud obtained in Example 1 or 2 above (… Figure 4 The depth information of each point in (a) is used to extract the plane in which each point is located. Each plane is then subjected to two-dimensional dilatation to obtain candidate planes. Figure 4 (b)); Based on the geometric relationships between the candidate planes, obtain the spatial model enclosed by the candidate planes. Figure 4 (c) in the middle.

[0059] The final dense depth map, after transformation, yields a dense point cloud of the target region. Subsequently, to avoid generating a large number of triangular meshes using traditional reconstruction methods, a planar extraction method is used to perform 3D reconstruction of the generated dense point cloud. It can be seen that the model reconstructed using the planar extraction method is simpler than the model generated by the triangular mesh method, making it more suitable for fields such as electromagnetic calculations.

[0060] This embodiment uses visual images and sparse point cloud images for modeling, which can make full use of the advantages of sparse point cloud images and visual images, reduce the technical requirements for point cloud images, and reduce modeling costs.

[0061] This embodiment uses a Transformer model instead of a traditional Convolutional Neural Network (CNN) to establish a depth estimation model. The model utilizes high-precision images to generate a dense, regular point cloud from the sparse point cloud acquired by LiDAR through upsampling. This generated point cloud is more conducive to the subsequent 3D reconstruction module. The modeling results are as follows... Figure 5 As shown, this fully demonstrates the effectiveness of the modeling method in this embodiment.

[0062] Example 4 discloses a training device for a fusion perception model based on visual images and point clouds, for use with the training method of Example 1. The device includes:

[0063] The model building module is used to build a Transformer model, which includes an encoder, a decoder, and an upsampling layer connected in series. The encoder includes a number of feature extraction layers, which contain a series of multi-head self-attention modules and multi-layer perceptron modules. Both the multi-head self-attention modules and multi-layer perceptron modules are connected to a LayerNorm layer before and a residual connection after. The decoder includes a number of decoding layers, which contain a series of two-dimensional convolutional modules and an upsampling module. The output of the decoder is then upsampled by a bilinear interpolation layer to restore the original image size, resulting in the final dense three-dimensional point cloud.

[0064] The sample generation and model training module is used to convert visual images and point cloud images into corresponding one-dimensional sequences, generate samples based on the one-dimensional sequences, and train the built Transformer model using the sample set composed of samples until the model converges, thus obtaining the fused perception model.

[0065] Example 5 discloses a dense 3D point cloud acquisition device based on visual images and point clouds, corresponding to Example 2. The device includes:

[0066] The acquisition module is used to acquire the visual image and point cloud image to be fused, convert them into data to be processed, and input them into the fusion perception model obtained in Example 1 to obtain the depth information of each pixel in the visual image to be fused.

[0067] The mapping module is used to map each pixel into the point cloud space based on the depth information of each pixel, so as to obtain a dense three-dimensional point cloud.

[0068] Example 6 discloses a modeling device based on visual images and point clouds, corresponding to Example 3. The device includes:

[0069] The extraction module is used to extract the plane where each point is located based on the depth information of each point in the dense three-dimensional point cloud obtained in Example 2.

[0070] The expansion module is used to perform two-dimensional expansion processing on each plane to obtain candidate planes;

[0071] The modeling module is used to obtain a spatial model enclosed by the candidate planes based on the geometric relationships between them.

Claims

1. A method for training a fusion perception model based on visual images and point clouds, characterized in that, Includes the following steps: A Transformer model is constructed, comprising an encoder, a decoder, and an upsampling layer connected in series. The encoder includes a number of feature extraction layers, each containing a multi-head self-attention module and a multi-layer perceptron module connected in series. Both the multi-head self-attention module and the multi-layer perceptron module are connected to a LayerNorm layer before and a residual connection after. The decoder includes a number of decoding layers, each containing a cascaded 2D convolutional module and an upsampling module. The output of the decoder is then upsampled using bilinear interpolation to restore the original image size, resulting in the final dense 3D point cloud. Visual images and point cloud images are converted into corresponding one-dimensional sequences, samples are generated based on the one-dimensional sequences, and the Transformer model is trained using the sample set composed of the samples until the model converges, thus obtaining the fusion perception model. The process of generating samples based on a one-dimensional sequence includes: One-dimensional sequences are concatenated to obtain two-dimensional concatenation results. The two-dimensional concatenation results are then trimmed into blocks, and finally each block is mapped to a one-dimensional space. The blocks mapped to one-dimensional space are mapped to D-dimensional block embedding information, and positional encoding is added to each block embedding information to obtain samples.

2. The training method for a fusion perception model based on visual images and point clouds according to claim 1, characterized in that, The multilayer perceptron module consists of two nonlinear layers connected in series with GELU.

3. The training method for a fusion perception model based on visual images and point clouds according to claim 1, characterized in that, A BatchNorm layer is provided between the 2D convolution module and the upsampling module.

4. The training method for a fusion perception model based on visual images and point clouds according to claim 1, characterized in that, The visual image and the point cloud image are data with the same shooting parameters, which include: shooting direction and shooting coordinates.

5. A method for acquiring dense 3D point clouds based on visual images and point clouds, characterized in that: After acquiring the visual image and point cloud image to be fused, they are converted into data to be processed and input into the fusion perception model obtained by any one of claims 1 to 4 above to obtain the depth information of each pixel in the visual image to be fused. Based on the depth information of each pixel, each pixel is mapped to the point cloud space to obtain a dense three-dimensional point cloud.

6. A modeling method based on visual images and point clouds, characterized in that: Based on the depth information of each point in the dense three-dimensional point cloud obtained by any one of claims 1 to 5, the plane in which each point is located is extracted, and each plane is subjected to two-dimensional expansion processing to obtain candidate planes; a spatial model closed by the candidate planes is obtained based on the geometric relationship between each candidate plane.

7. A training device for a fusion perception model based on visual images and point clouds, for use with any one of the training methods of claims 1 to 4, characterized in that, The device includes: The model building module is used to build a Transformer model, which includes an encoder, a decoder, and an upsampling layer connected in series. The encoder includes a number of feature extraction layers, which contain a series of multi-head self-attention modules and multi-layer perceptron modules. Both the multi-head self-attention modules and multi-layer perceptron modules are connected to a LayerNorm layer before and a residual connection after. The decoder includes a number of decoding layers, which contain a series of two-dimensional convolutional modules and an upsampling module. The output of the decoder is then upsampled by a bilinear interpolation layer to restore the original image size, resulting in the final dense three-dimensional point cloud. The sample generation and model training module is used to convert visual images and point cloud images into corresponding one-dimensional sequences, generate samples based on the one-dimensional sequences, and train the built Transformer model using the sample set composed of samples until the model converges, thus obtaining the fusion perception model. The process of generating samples based on a one-dimensional sequence includes: One-dimensional sequences are concatenated to obtain two-dimensional concatenation results. The two-dimensional concatenation results are then trimmed into blocks, and finally each block is mapped to a one-dimensional space. The blocks mapped to one-dimensional space are mapped to D-dimensional block embedding information, and positional encoding is added to each block embedding information to obtain samples.

8. A device for acquiring dense 3D point clouds based on visual images and point clouds, characterized in that, The device includes: The acquisition module is used to acquire the visual image and point cloud image to be fused, convert them into data to be processed, and input them into the fusion perception model obtained by any one of claims 1 to 4 above, so as to obtain the depth information of each pixel in the visual image to be fused. The mapping module is used to map each pixel into the point cloud space based on the depth information of each pixel, so as to obtain a dense three-dimensional point cloud.

9. A modeling apparatus based on visual images and point clouds, characterized in that, The device includes: The extraction module is used to extract the plane where each point is located based on the depth information of each point in the dense three-dimensional point cloud obtained by any one of claims 1 to 5. The expansion module is used to perform two-dimensional expansion processing on each plane to obtain candidate planes; The modeling module is used to obtain a spatial model enclosed by the candidate planes based on the geometric relationships between them.

Images