3D point cloud reconstruction method and device, electronic equipment and storage medium

Abstract

The invention provides a 3D point cloud reconstruction method and device, electronic equipment and a storage medium, and is applied to a preset point cloud reconstruction model, and the method comprises the steps: extracting a target feature tensor corresponding to a two-dimensional input image through an encoder; performing 3D voxel conversion processing of different roughness degrees on the target feature tensor by using a preset decoder to obtain 3D voxels of different roughness degrees; performing 3D reconstruction adaptively based on the 3D voxels with different roughness degrees by using a context sensing fusion module to obtain fused 3D voxels; and based on the refinement device, optimizing and fusing the 3D voxels through jump connection, and generating a three-dimensional reconstruction result corresponding to the two-dimensional input image. According to the invention, the Swin Transform network model is used as an encoder for 2D image feature extraction in the preset point cloud reconstruction model, the 2D image feature extraction capability is enhanced, and the accuracy of subsequent point cloud reconstruction is improved.

Description

Technical Field

[0001] The present application relates to the technical field of 3D point cloud reconstruction, and in particular to a 3D point cloud reconstruction method, device, electronic device and storage medium. Background Art

[0002] The Pix2Vox network model uses the older and simpler VGG16 model in the 2D image feature extraction part. The VGG16 model can achieve good feature extraction results for simple scenes, but it may be difficult to achieve ideal results when encountering more complex scenes, such as urban public roads and indoor warehouse environments. It is impossible to fully and comprehensively extract 2D image features, resulting in unsatisfactory results for 3D point cloud reconstruction based on the extracted features. Summary of the invention

[0003] In view of this, the purpose of the present application is to provide at least a 3D point cloud reconstruction method, device, electronic device and storage medium, by using the Swin Transformer network model as an encoder for 2D image feature extraction in the entire Pix2Vox point cloud reconstruction model, to enhance the 3D image feature extraction capability and improve the accuracy of subsequent point cloud reconstruction.

[0004] This application mainly includes the following aspects:

[0005] In a first aspect, an embodiment of the present application provides a 3D point cloud reconstruction method, which is applied to a preset point cloud reconstruction model. The preset point cloud reconstruction model includes an encoder, a decoder, a context-aware fusion module and a refiner. The encoder is constructed by a SwinTransformer network model. The method includes: using the encoder to extract a target feature map corresponding to a two-dimensional input image; using the decoder to perform 3D voxel conversion processing with different degrees of roughness on the target feature map to obtain 3D voxels with different degrees of roughness; using the context-aware fusion module to adaptively perform 3D reconstruction based on 3D voxels with different degrees of roughness to obtain fused 3D voxels; based on the refiner, the fused 3D voxels are optimized through jump connections to generate a three-dimensional reconstruction result corresponding to the two-dimensional input image.

[0006] In one possible implementation, the Swin Transformer network model includes a first-stage processing module, a second-stage processing module, a third-stage processing module, and a fourth-stage processing module, wherein a target feature map corresponding to a two-dimensional input image is extracted in the following manner: the two-dimensional input image is input into the first-stage processing module to obtain a first feature extraction map; the first feature extraction map is input into the second-stage processing module to obtain a second feature extraction map; the second feature extraction map is input into the third-stage processing module to obtain a third feature extraction map; the third feature extraction map is input into the third-stage processing module to obtain a target feature map.

[0007] In one possible implementation, the first stage processing module includes a block division layer, a first linear embedding layer and a first feature network extraction layer, the first feature network extraction layer includes a plurality of Swin Transformer feature extraction networks connected in series, wherein the first stage processing module obtains a first feature extraction graph in the following manner: the two-dimensional input image is input into the block division layer for segmentation processing to obtain a plurality of first image blocks with an image size of a first preset size and a feature dimension of a first feature dimension; each first image block is input into the first linear embedding layer for dimensional transformation processing to obtain a first feature vector corresponding to each first image block, and the feature dimension corresponding to the first feature vector is a second feature dimension; for each first feature vector, the feature vector is input into the first feature network extraction layer for feature extraction to obtain a first output image block corresponding to the first feature vector, and the image size corresponding to the first output image block is a first preset size and the feature dimension is a second feature dimension; a first feature extraction graph is formed by the plurality of first output image blocks.

[0008] In one possible implementation, the second stage processing module includes a first fusion layer, a second linear embedding layer and a second feature network extraction layer, and the second feature network extraction layer includes a plurality of Swin Transformer networks connected in series; wherein the second stage processing module determines the second feature extraction map in the following manner: inputting the first feature extraction map into the first fusion layer to merge adjacent 2×2 first output picture blocks in the first feature extraction map to obtain a plurality of second picture blocks, the picture size corresponding to the second picture blocks is a second preset size, and the feature dimension is a third feature dimension; inputting the plurality of second picture blocks into the second linear layer for dimensionality reduction processing to obtain a second feature vector corresponding to each second picture block, the feature dimension corresponding to the second feature vector is half of the third feature dimension, and the picture size is the second preset size; inputting the plurality of second feature vectors into the second feature network extraction layer to obtain a plurality of second output picture blocks, the picture size and feature dimension corresponding to the second output picture blocks are the same as the second feature vectors; and forming a second feature extraction map by the plurality of second output picture blocks.

[0009] In one possible implementation, the third stage processing module includes a second fusion layer, a second linear embedding layer and a third feature network extraction layer, and the third feature network extraction layer includes a plurality of Swin Transformer networks connected in series; wherein the third stage processing module determines the third feature extraction map in the following manner: inputting the second feature extraction map into the second fusion layer to merge the adjacent 2×2 second output picture blocks in the second feature extraction map to obtain a plurality of third picture blocks, the picture size corresponding to the third picture blocks is a third preset size, and the feature dimension is a fourth feature dimension; inputting the plurality of third picture blocks into the third linear embedding layer for dimensionality reduction processing to obtain a third feature vector corresponding to each third picture block, the feature dimension corresponding to the third feature vector is half of the fourth feature dimension, and the picture size is a third preset size; inputting the plurality of third feature vectors into the third feature network extraction layer to obtain a plurality of third output picture blocks, the picture size and feature dimension corresponding to the third output picture blocks are the same as the third feature vectors; and forming a third feature extraction map by the plurality of third output picture blocks.

[0010] In one possible implementation, the fourth stage processing module includes a third fusion layer, a fourth linear embedding layer and a fourth feature network extraction layer, the fourth feature network extraction layer includes a plurality of Swin Transformer networks connected in series, wherein the fourth stage processing module determines the target feature map in the following manner: inputting the third feature extraction map into the third fusion layer to merge the adjacent 2×2 third output picture blocks in the third feature extraction map to obtain a plurality of fourth picture blocks, the picture size corresponding to the fourth picture blocks is a fourth preset size, and the feature dimension is a fifth feature dimension; inputting the plurality of fourth picture blocks into the fourth linear embedding layer for dimensionality reduction processing to obtain a fourth feature vector corresponding to each fourth picture block, the feature dimension corresponding to the fourth feature vector is half of the fifth feature dimension, and the picture size is a fourth preset size; inputting the plurality of fourth feature vectors into the fourth feature network extraction layer to obtain a plurality of fourth output picture blocks, the picture size and feature dimension corresponding to the fourth output picture blocks are the same as the fourth feature vector; and forming the target feature map by the plurality of fourth output picture blocks.

[0011] In one possible implementation, the Swin Transformer network includes at least two normalization layers, a self-attention calculation layer and a multi-layer perceptron, wherein the normalization layers are respectively arranged before the self-attention calculation layer and the multi-layer perceptron, wherein the Swin Transformer network obtains a first output image block in the following manner: inputting a first feature vector into one of the normalization layers, performing normalization processing with the first feature vector, and obtaining a first feature vector after normalization processing; inputting the first feature vector after normalization processing into the self-attention calculation layer, performing self-attention calculation within a non-overlapping window, capturing context feature information, and obtaining a feature processing vector; inputting the feature processing vector into another normalization layer, and obtaining a feature processing vector after normalization processing; inputting the feature processing vector after normalization processing into the multi-layer perceptron for feature transformation, and obtaining the corresponding first output image block.

[0012] In the second aspect, an embodiment of the present application also provides a 3D point cloud reconstruction device, which is applied to a preset point cloud reconstruction model. The preset point cloud reconstruction model includes an encoder, a decoder, a context-aware fusion module and a refiner. The encoder is built by a SwinTransformer network model, and the device includes: a feature extraction module, which is used to use the encoder to extract a target feature map corresponding to a two-dimensional input image; a conversion module, which is used to use the decoder to perform 3D voxel conversion processing on the target feature map with different degrees of roughness to obtain 3D voxels with different degrees of roughness; a voxel fusion module, which is used to use the context-aware fusion module to adaptively perform 3D reconstruction based on 3D voxels with different degrees of roughness to obtain fused 3D voxels; a reconstruction module, which is used to optimize the fused 3D voxels through jump connections based on the refiner to generate a three-dimensional reconstruction result corresponding to the two-dimensional input image.

[0013] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the 3D point cloud reconstruction method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of 3D point cloud reconstruction in the above-mentioned first aspect or any possible implementation of the first aspect are performed.

[0015] A 3D point cloud reconstruction method, device, electronic device and storage medium provided in an embodiment of the present application are applied to a Pix2Vox point cloud reconstruction model. The Pix2Vox point cloud reconstruction model includes an encoder, a decoder, a context-aware fusion module and a refiner. The encoder is constructed by a Swin Transformer network model. The method includes: using the encoder to extract a target feature tensor corresponding to a two-dimensional input image; using a preset decoder to perform 3D voxel conversion processing of the target feature tensor with different degrees of roughness to obtain 3D voxels with different degrees of roughness; using a context-aware fusion module to adaptively perform 3D reconstruction based on 3D voxels with different degrees of roughness to obtain fused 3D voxels; based on the refiner, the fused 3D voxels are optimized through jump connections to generate a three-dimensional reconstruction result corresponding to the two-dimensional input image. The present application uses the Swin Transformer network model as an encoder for 2D image feature extraction in the entire Pix2Vox point cloud reconstruction model to enhance the 3D image feature extraction capability and improve the accuracy of subsequent point cloud reconstruction.

[0016] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flow chart of a 3D point cloud reconstruction method provided in an embodiment of the present application is shown;

[0019] Figure 2 A structural diagram of a Swin Transformer network model processing flow provided in an embodiment of the present application is shown;

[0020] Figure 3 A schematic diagram of a feature extraction structure provided in an embodiment of the present application is shown;

[0021] Figure 4 A functional module diagram of a 3D point cloud reconstruction device provided in an embodiment of the present application is shown;

[0022] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0024] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0025] In the prior art, the Pix2Vox network model is generally used to reconstruct 3D point clouds from 2D images. The Pix2Vox network model includes an encoder, a decoder, a context-aware fusion module and a refiner. The encoder in the Pix2Vox network model provided by the prior art mainly uses the first nine convolutional layers of the VGG16 model pre-trained on the ImageNet open source dataset to extract features from 2D images. The VGG16 model may achieve good feature extraction effects for simple scenes, but when extracting features for more complex scenes, such as urban public roads, indoor warehouse environments, etc., it is difficult to fully extract 2D image features, thereby affecting the subsequent reconstruction of 3D point clouds.

[0026] Based on this, the embodiments of the present application provide a 3D point cloud reconstruction method, device, electronic device and storage medium, which uses the Swin Transformer network model as an encoder for 2D image feature extraction in a preset point cloud reconstruction model to enhance the 2D image feature extraction capability and improve the accuracy of subsequent point cloud reconstruction, as follows:

[0027] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a 3D point cloud reconstruction method provided by an embodiment of the present application. Figure 1As shown, the method provided in the embodiment of the present application is applied to a preset point cloud reconstruction model, and the preset point cloud reconstruction model includes an encoder, a decoder, a context-aware fusion module and a refiner. The preset point cloud reconstruction model in the present application is an ST-Pix2Vox network model improved on the basis of the Pix2Vox model. Specifically, the encoder in the preset point cloud reconstruction ST-Pix2Vox is constructed by the Swin Transformer network model.

[0028] Based on the preset point cloud reconstruction model provided in this application, the method provided in this application includes the following steps:

[0029] S100, using an encoder to extract a target feature map corresponding to a two-dimensional input image.

[0030] S200 , using a decoder to perform 3D voxel conversion processing of different coarseness levels on the target feature map to obtain 3D voxels of different coarseness levels.

[0031] S300 , using a context-aware fusion module, adaptively performing 3D reconstruction based on 3D voxels of different coarseness levels to obtain fused 3D voxels.

[0032] S400, based on the refiner, optimizing and fusing 3D voxels through skip connections to generate a three-dimensional reconstruction result corresponding to the two-dimensional input image.

[0033] In a specific implementation, in steps S100 to S400 of the present application, the method provided by the present application is applied to a preset network reconstruction model ST-Pix2Vox. In the preset network reconstruction model ST-Pix2Vox, a Swin Transformer network model is used to replace the VGG16 network model in the original Pix2Vox network model as an encoder for 2D image feature extraction. The Swin Transformer network model is a new type of visual Transformer model. The core feature of the Swin Transformer network model is to achieve efficient self-attention calculation through a hierarchical structure and a moving window mechanism. Specifically, the Swin Transformer network model is used as an encoder in the preset network reconstruction model ST-Pix2Vox to extract features from two-dimensional input images, thereby enhancing the feature extraction capability of two-dimensional input images and improving the accuracy of subsequent point cloud reconstruction.

[0034] Specifically, the Swin Transformer network model has the following advantages over the CNN (convolutional neural network) in the VGG16 network model in terms of feature extraction of two-dimensional input images:

[0035] ① Global information attention: The Swin Transformer network model has the advantage of global information attention when processing sequence data, which means that each position in the two-dimensional input image can directly pay attention to the information of all other positions, which is more advantageous for feature extraction such as image classification or cross-image tasks.

[0036] ② Scalability: The Swin Transformer network model can be more easily expanded to larger image sizes, while CNN may face challenges in memory and computing resources when processing large-size images.

[0037] ③ Position encoding: Swin Transformer network models use position encoding to process position information in sequences, which makes them more flexible for processing images of different sizes.

[0038] ④Parallel computing: In some cases, the Swin Transformer network model can better take advantage of parallel computing, especially when processing large images.

[0039] In a preferred embodiment, the Swin Transformer network model includes a first-stage processing module, a second-stage processing module, a third-stage processing module, and a fourth-stage processing module.

[0040] Preferably, the hierarchical structure of the Swin Transformer network model is divided into four stage processing modules, including a first stage processing module, a second stage processing module, a third stage processing module and a fourth stage processing module. Different stage processing modules correspond to different feature network extraction layers, and different feature network extraction layers correspond to different numbers of Swin Transformer networks. The input of each stage processing module is the feature map generated by the previous stage processing module.

[0041] Among them, the target feature map corresponding to the two-dimensional input image is extracted in the following way:

[0042] The two-dimensional input image is input into the first stage processing module to obtain a first feature extraction map, the first feature extraction map is input into the second stage processing module to obtain a second feature extraction map, the second feature extraction map is input into the third stage processing module to obtain a third feature extraction map, and the third feature extraction map is input into the third stage processing module to obtain a target feature map.

[0043] Specifically,

[0044] Preferably, the first stage processing module includes a block partitioning layer, a first linear embedding layer and a first feature network extraction layer, and the first feature network extraction layer includes a plurality of Swin Transformer feature extraction networks connected in series.

[0045] In a preferred embodiment, the first stage processing module obtains the first feature extraction graph by:

[0046] The two-dimensional input image input block is divided into layers for segmentation processing to obtain multiple first image blocks with a picture size of a first preset size and a feature dimension of a first feature dimension. Each first image block is input into a first linear embedding layer for dimension transformation processing to obtain a first feature vector corresponding to each first image block, and the feature dimension corresponding to the first feature vector is a second feature dimension. For each first feature vector, the feature vector is input into a first feature network extraction layer for feature extraction to obtain a first output image block corresponding to the first feature vector, and the picture size corresponding to the first output image block is a first preset size and the feature dimension is a second feature dimension. A first feature extraction graph is formed by the multiple first output image blocks.

[0047] See also Figure 2 , Figure 2 The following is a schematic diagram showing a Swin Transformer network model processing flow provided in an embodiment of the present application. Figure 2 As shown, the specification of the two-dimensional input image is H×W×3, H×W represents the image size corresponding to the two-dimensional input image, wherein H represents the image height corresponding to the two-dimensional input image, W represents the image width corresponding to the two-dimensional input image, and 3 represents the number of channels corresponding to the two-dimensional input image.

[0048] In the first stage processing module Stage1 corresponding to the Swin Transformer network model, the two-dimensional input image is divided into multiple first image patches of image size 4×4 through the block partitioning layer. The first feature dimension corresponding to each first image patch is 4×4×3=48, so there are a total of First image blocks are obtained, and then each first image block patch is transformed into a first feature vector with a feature dimension of a second feature dimension C (C represents an arbitrary dimension) through a first linear embedding layer. Then, the first feature network extraction layer is applied to perform self-attention calculation in non-overlapping windows to capture local context information, and the pixels inside the image sub-image corresponding to each first feature vector are modeled. The image sub-image is nonlinearly transformed through a multi-layer perceptron (MLP) in the first feature network extraction layer to obtain multiple first output image blocks. The image size corresponding to each first output image block is 4×4, and the second feature dimension corresponding to each first output image block is 4×4×C. A first feature extraction map is formed by multiple first output image blocks, and the size of the first feature extraction map is

[0049]

[0050] The Swin Transformer network includes at least two normalization layers, a self-attention calculation layer, and a multi-layer perceptron. The normalization layer is respectively arranged before the self-attention calculation layer and the multi-layer perceptron. The Swin Transformer network obtains the first output image block in the following manner:

[0051] The first feature vector is input into one of the standardization layers, and the first feature vector is standardized to obtain the standardized first feature vector. The standardized first feature vector is input into the self-attention calculation layer to perform self-attention calculation in a non-overlapping window to capture context feature information and obtain a feature processing vector. The feature processing vector is input into another standardization layer to obtain a standardized feature processing vector. The standardized feature processing vector is input into the multi-layer perceptron for feature transformation to obtain the corresponding first output image block.

[0052] The feature network extraction layer corresponding to the processing modules at different stages includes at least one feature extraction structure connected in series, each feature extraction structure includes two Swin Transformer feature extraction networks connected in series, and the self-attention calculation layer includes a first self-attention calculation layer W-MSA and a second self-attention calculation layer SW-MSA. Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a feature extraction structure provided in an embodiment of the present application. Figure 3 The feature network extraction structure shown includes two first Swin Transformer feature extraction networks a and second Swin Transformer feature extraction networks b connected in series, and the output of the first Swin Transformer feature extraction network a is connected to the input of the second Swin Transformer feature extraction network b.

[0053] Preferably, the first Swin Transformer feature extraction network a includes: a first normalization layer LayerNorm1 (LN1), a first self-attention calculation layer W-MSA, a second normalization layer LayerNorm2 (LN2) and a first multi-layer perceptron MLP1.

[0054] Among them, it is assumed that the input feature vector corresponding to the feature extraction structure is x l-1 , then x l-1 Input the first normalization layer LN1 to obtain the normalized feature vector, and input the normalized feature vector into the first self-attention calculation layer W-MSA. W-MSA divides the feature vector output by LN1 into non-overlapping windows of fixed size. Self-attention is calculated separately in each window without involving cross-window information interaction, and the processed feature vector is obtained. Will Input the second normalization layer LN2 for normalization again, and input the normalized feature vector into the first multi-layer perceptron MLP1 for feature transformation to obtain the feature vector x l The multilayer perceptron includes two fully connected layers and a GELU activation function, which transforms the feature vector x l Input the second Swin Transformer feature extraction network b.

[0055] Preferably, the second Swin Transformer feature extraction network b includes a third normalization layer LayerNorm3 (LN3), a second self-attention calculation layer SW-MSA, a fourth normalization layer LayerNorm4 (LN4) and a second multi-layer perceptron MLP2.

[0056] Specifically, x l Input the third normalization layer LN3 to obtain the normalized feature vector, and input the normalized feature vector into the second self-attention calculation layer SW-MSA. Through SW-MSA, the window position is moved on the feature vector of every other layer (usually by half the size of the window) to form a new window division, and the self-attention is calculated in the moved window to finally obtain the processed feature vector. SW-MSA allows the window to cover the area not covered by the previous layer, thereby realizing cross-window information interaction. By moving the position of the window, SW-MSA can capture a wider range of contextual information and enhance the global representation capability of the features while still maintaining computational efficiency.

[0057] Will The fourth normalization layer LN4 is input for normalization again, and the normalized feature vector is input into the second multi-layer perceptron MLP2 for feature transformation to obtain the feature vector x l+1.

[0058] Preferably, the Swin Transformer feature extraction network also includes multiple residual connection layers, and a residual connection layer is connected after each layer in the Swin Transformer feature extraction network for performing residual calculation to alleviate the gradient vanishing problem of the deep network.

[0059] In a preferred embodiment, the second stage processing module includes a first fusion layer, a second linear embedding layer, and a second feature network extraction layer, the second feature network extraction layer includes a plurality of Swin Transformer networks connected in series, wherein the second stage processing module determines the second feature extraction map in the following manner:

[0060] The first feature extraction map is input into the first fusion layer to merge the adjacent 2×2 first output picture blocks in the first feature extraction map to obtain multiple second picture blocks, the picture size corresponding to the second picture block is the second preset size = 2×the first preset size, and the feature dimension is the third feature dimension = 4×the second feature dimension, the multiple second picture blocks are input into the second linear layer for dimensionality reduction processing to obtain a second feature vector corresponding to each second picture block, the feature dimension corresponding to the second feature vector is half of the third feature dimension, and the picture size is the second preset size, the multiple second feature vectors are input into the second feature network extraction layer to obtain multiple second output picture blocks, the picture size and feature dimension corresponding to the second output picture block are the same as the second feature vector, and the second feature extraction map is formed by the multiple second output picture blocks.

[0061] In the present application, a fusion layer is introduced into the second-stage processing module, the third-stage processing module, and the fourth-stage processing module to merge features, thereby reducing the number of tokens and increasing the dimension of the feature map.

[0062] In a specific example, through the first fusion layer, the adjacent 2x2 first output modules of the first feature extraction map are merged to obtain a merged second picture block, the picture size corresponding to the second picture block is 8×8, and the feature dimension becomes 2×2×C, and then the second picture block is reduced to 2C through the second linear embedding layer to obtain the second feature vector corresponding to each second picture block. In the second feature network extraction layer, the second feature vector is processed by two Swin Transformer networks connected in series to obtain the corresponding second output picture block. The picture size corresponding to the second output picture block is 8×8 and the feature dimension is 2C, that is, the size of the second feature extraction map formed by multiple second output pictures is

[0063] In a preferred embodiment, the third stage processing module includes a second fusion layer, a second linear embedding layer and a third feature network extraction layer, the third feature network extraction layer includes a plurality of Swin Transformer networks connected in series, wherein the third stage processing module determines the third feature extraction map in the following manner:

[0064] The second feature extraction map is input into the second fusion layer to merge the adjacent 2×2 second output picture blocks in the second feature extraction map to obtain multiple third picture blocks, the picture size corresponding to the third picture blocks is a third preset size, and the feature dimension is a fourth feature dimension, the multiple third picture blocks are input into the third linear embedding layer for dimensionality reduction processing to obtain a third feature vector corresponding to each third picture block, the feature dimension corresponding to the third feature vector is half of the fourth feature dimension, and the picture size is the third preset size, the multiple third feature vectors are input into the third feature network extraction layer to obtain multiple third output picture blocks, the picture size and feature dimension corresponding to the third output picture block are the same as the third feature vector, and a third feature extraction map is formed by the multiple third output picture blocks.

[0065] In a specific example, after determining the second feature extraction map, the second feature extraction map is input into the second fusion layer, and the adjacent 2x2 second output picture blocks are merged through the second fusion layer to obtain multiple third picture blocks. The picture size corresponding to the third picture block is 16×16, and the feature dimension becomes 2×2×2C. Then, the third picture block is reduced to 4C through the third linear embedding layer to obtain a third feature vector corresponding to each third picture block. In the third feature network extraction layer, the third feature vector is processed through 6 Swin Transformer networks (3 feature extraction structures) connected in series to obtain the corresponding third output picture block. The picture size corresponding to the third output picture block is 16×16 and the feature dimension is 4C, that is, the size of the third feature extraction map formed by multiple third output pictures is

[0066] In the third-stage processing module, six Swin Transformer networks are applied to enhance the global context understanding ability of features through a larger range of moving window self-attention mechanism, and to extract higher-level semantic features through deep nonlinear transformations.

[0067] In a preferred embodiment, the fourth stage processing module includes a third fusion layer, a fourth linear embedding layer and a fourth feature network extraction layer, and the fourth feature network extraction layer includes a plurality of Swin Transformer networks connected in series, wherein the fourth stage processing module determines the target feature map by:

[0068] The third feature extraction map is input into the third fusion layer to merge the adjacent 2×2 third output picture blocks in the third feature extraction map to obtain multiple fourth picture blocks, the picture size corresponding to the fourth picture block is a fourth preset size = 2×third preset size, and the feature dimension is a fifth feature dimension, the multiple fourth picture blocks are input into the fourth linear embedding layer for dimensionality reduction processing to obtain a fourth feature vector corresponding to each fourth picture block, the feature dimension corresponding to the fourth feature vector is half of the fifth feature dimension, and the picture size is a fourth preset size, the multiple fourth feature vectors are input into the fourth feature network extraction layer to obtain multiple fourth output picture blocks, the picture size and feature dimension corresponding to the fourth output picture block are the same as the fourth feature vector, and the third feature extraction map is formed by the multiple fourth output picture blocks.

[0069] In a specific example, after determining the third feature extraction map, the three feature extraction maps are input into the third fusion layer, and the adjacent 2x2 third output picture blocks are merged through the third fusion layer to obtain multiple fourth picture blocks. The image size corresponding to the fourth picture block is 32×32, and the feature dimension becomes 2×2×4C. Then, the fourth picture block is reduced to 8C through the fourth linear embedding layer to obtain a fourth feature vector corresponding to each fourth picture block. In the fourth feature network extraction layer, the fourth feature vector is processed through two Swin Transformer networks (1 feature extraction structure) connected in series to obtain the corresponding fourth output picture block. The image size corresponding to the fourth output picture block is 32×32 and the feature dimension is 8C, that is, the target feature map size formed by multiple fourth output pictures is

[0070]

[0071] In the fourth stage processing module, two Swin Transformer networks are used to further enhance the feature output representation, making full use of the global and detail information of the image to provide rich, high-level target feature maps for downstream tasks (2D to 3D).

[0072] The Swin Transformer network model includes different versions (Swin-T, Swin-S, Swin-B, Swin-L). The model parameters corresponding to the internal modules of different versions of the Swin Transformer network model are different. In this application, the user can adjust the model parameters corresponding to the processing of each stage of the model based on the complexity of the 3D point cloud reconstruction calculation to switch the different versions corresponding to the Swin Transformer network model, thereby achieving different performance and computational complexity. The complexity of Swin-T, Swin-S, Swin-B, and Swin-L increases in sequence.

[0073] Please refer to Table 1 below, which shows the model parameters corresponding to the processing modules at different stages of different versions of the Swin Transformer network model:

[0074] Table 1

[0075]

[0076]

[0077] As shown in Table 1, C represents the number of channels of the feature map, L represents the number of Swin Transformer network models contained in each stage processing module, L is an even number, and the total number of parameters represents the number of parameters of the entire Swin Transformer network model, in million (M). The total number of parameters is closely related to the computational complexity and performance of the Swin Transformer network model. In the Swin-T version of the Swin Transformer network model, the second feature dimension C corresponding to the first stage processing module Stage1 is 96, and the corresponding number of Swin Transformer feature extraction network is L1=2. The feature dimension 2C=192 corresponding to the second feature extraction map output by the second stage processing module Stage2, and the corresponding number of Swin Transformer feature extraction network is L2=2. The feature dimension 4C=384 corresponding to the third feature extraction map output by the third stage processing module Stage3, and the corresponding number of Swin Transformer feature extraction network is L3=6. The feature dimension 8C=768 corresponding to the target feature map output by the fourth stage processing module Stage4, and the corresponding number of Swin Transformer feature extraction network is L4=2. The Swin-T version of Swin The total number of parameters corresponding to the Transformer network model is 29M. Other versions are similar to the Swin-T version and will not be described here.

[0078] In a specific embodiment, in step S200, after obtaining the target feature map, the 2D target feature map is converted into 3D voxels of different coarseness through a decoder. Specifically, the decoder mainly converts the target feature map into 3D voxels step by step through five transposed convolutional layers, wherein the parameters of the five transposed convolutional layers are set as follows: the convolution kernel size of the first four transposed convolutional layers is 4×4, the step size is 2, and the padding is 1; the convolution kernel size of the last transposed convolutional layer is 1×1, which is used to further refine the voxels; the number of output channels of each transposed convolutional layer gradually decreases to adapt to the conversion process from 2D feature maps to 3D voxels; the number of output channels of the last transposed convolutional layer is typically 1, corresponding to the generated 3D voxels.

[0079] Each transposed convolutional layer is followed by a batch normalization layer and an activation function to speed up the training process and enhance the expressiveness of the network.

[0080] The last transposed convolution layer of the decoder is a transposed convolution layer with a Sigmoid function. The last transposed convolution layer is used to generate the final 3D voxel output. The Sigmoid function limits the output between 0 and 1, which is suitable for voxel reconstruction tasks.

[0081] The U-net connection is used in the decoder provided in this application to connect the feature maps corresponding to the encoder and the decoder to preserve the local structure in the fused voxels. This connection method helps to improve the accuracy of reconstruction.

[0082] In step S300, the context-aware fusion module can adaptively select high-quality reconstruction of each part from different coarse 3D voxels to obtain a fused reconstruction of the entire object. The context-aware fusion module uses context to perform 3D reconstruction across multiple views, which is the first time that multi-view context is used for 3D reconstruction.

[0083] In step S400, a refiner further optimizes the fused 3D voxels through skip connections to generate a final reconstruction result. The refiner is used to improve the accuracy and quality of reconstruction.

[0084] Based on the same application concept, the embodiments of the present application also provide a 3D point cloud reconstruction device corresponding to the 3D point cloud reconstruction method provided in the above embodiments. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the 3D point cloud reconstruction method in the above embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0085] See also Figure 4 , Figure 4 FIG. 1 shows a functional module diagram of a 3D point cloud reconstruction device provided in an embodiment of the present application. Figure 4 As shown, the 3D point cloud reconstruction device is applied to a preset point cloud reconstruction model, the preset point cloud reconstruction model includes an encoder, a decoder, a context-aware fusion module and a refiner, the encoder is formed by a Swin Transformer network model, and the 3D point cloud reconstruction device includes:

[0086] A feature extraction module 500 is used to extract a target feature map corresponding to a two-dimensional input image using an encoder;

[0087] The conversion module 510 is used to perform 3D voxel conversion processing of different coarseness on the target feature map by using the decoder to obtain 3D voxels of different coarseness;

[0088] A voxel fusion module 520, configured to adaptively perform 3D reconstruction based on 3D voxels of different coarseness levels using a context-aware fusion module to obtain fused 3D voxels;

[0089] The reconstruction module 530 is used to generate a three-dimensional reconstruction result corresponding to the two-dimensional input image by optimizing and fusing 3D voxels through skip connections based on the refiner.

[0090] Based on the same application idea, please refer to Figure 5 , Figure 5 The schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. The electronic device 600 includes: a processor 610, a memory 620 and a bus 630, the memory 620 stores machine-readable instructions executable by the processor 610, when the electronic device 600 is running, the processor 610 and the memory 620 communicate through the bus 630, and the machine-readable instructions are executed by the processor 610 when running to perform the steps of the 3D point cloud reconstruction method provided in any of the above embodiments.

[0091] Based on the same application concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the 3D point cloud reconstruction method provided in the above embodiment are executed.

[0092] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0093] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0094] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0095] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0096] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A 3D point cloud reconstruction method, characterized in that: Applied to a preset point cloud reconstruction model, the preset point cloud reconstruction model includes an encoder, a decoder, a context-aware fusion module and a refiner, the encoder is formed by a Swin Transformer network model, Wherein, the method comprises: Extracting a target feature map corresponding to a two-dimensional input image using the encoder; Using a decoder to perform 3D voxel conversion processing of different coarseness on the target feature map to obtain 3D voxels of different coarseness; Using the context-aware fusion module, 3D reconstruction is adaptively performed based on 3D voxels of different coarseness to obtain fused 3D voxels; The fused 3D voxels are optimized based on the refiner through skip connections to generate a three-dimensional reconstruction result corresponding to the two-dimensional input image.

2. The method according to claim 1, characterized in that The Swin Transformer network model includes a first-stage processing module, a second-stage processing module, a third-stage processing module and a fourth-stage processing module. Among them, the target feature map corresponding to the two-dimensional input image is extracted in the following way: Inputting the two-dimensional input image into the first stage processing module to obtain a first feature extraction map; Inputting the first feature extraction map into the second stage processing module to obtain a second feature extraction map; Inputting the second feature extraction map into the third stage processing module to obtain a third feature extraction map; The third feature extraction map is input into the third stage processing module to obtain the target feature map.

3. The method according to claim 2, characterized in that The first stage processing module includes a block partitioning layer, a first linear embedding layer and a first feature network extraction layer, wherein the first feature network extraction layer includes a plurality of SwinTransformer feature extraction networks connected in series, Wherein, the first stage processing module obtains the first feature extraction graph in the following manner: Inputting the two-dimensional input image into the block division layer for segmentation processing to obtain a plurality of first image blocks with an image size of a first preset size and a feature dimension of a first feature dimension; Input each first image block into the first linear embedding layer for dimension transformation processing to obtain a first feature vector corresponding to each first image block, where the feature dimension corresponding to the first feature vector is the second feature dimension; For each first feature vector, input the feature vector into the first feature network extraction layer for feature extraction, and obtain a first output picture block corresponding to the first feature vector, wherein the picture size corresponding to the first output picture block is a first preset size, and the feature dimension is a second feature dimension; The first feature extraction graph is formed by multiple first output image blocks.

4. The method according to claim 2, characterized in that: The second stage processing module includes a first fusion layer, a second linear embedding layer, and a second feature network extraction layer, and the second feature network extraction layer includes a plurality of Swin Transformer networks connected in series; Wherein, the second stage processing module determines the second feature extraction graph in the following manner: Inputting the first feature extraction image into a first fusion layer to merge adjacent 2×2 first output image blocks in the first feature extraction image to obtain a plurality of second image blocks, wherein the image size corresponding to the second image blocks is a second preset size, and the feature dimension is a third feature dimension; Input the multiple second image blocks into the second linear layer for dimensionality reduction processing to obtain a second feature vector corresponding to each second image block, wherein the feature dimension corresponding to the second feature vector is half of the third feature dimension, and the image size is a second preset size; Inputting a plurality of second feature vectors into the second feature network extraction layer to obtain a plurality of second output picture blocks, wherein the picture size and feature dimension corresponding to the second output picture blocks are the same as those of the second feature vectors; The second feature extraction map is formed by multiple second output picture blocks.

5. The method according to claim 2, characterized in that: The third stage processing module includes a second fusion layer, a third linear embedding layer and a third feature network extraction layer, and the third feature network extraction layer includes a plurality of Swin Transformer networks connected in series; Wherein, the third stage processing module determines the third feature extraction graph in the following manner: Input the second feature extraction map into the second fusion layer to merge adjacent 2×2 second output image blocks in the second feature extraction map to obtain a plurality of third image blocks, wherein the image size corresponding to the third image blocks is a third preset size, and the feature dimension is a fourth feature dimension; Inputting the multiple third image blocks into the third linear embedding layer for dimensionality reduction processing to obtain a third eigenvector corresponding to each third image block, wherein the feature dimension corresponding to the third eigenvector is half of the fourth feature dimension, and the image size is a third preset size; Inputting a plurality of third feature vectors into the third feature network extraction layer to obtain a plurality of third output picture blocks, wherein the picture size and feature dimension corresponding to the third output picture blocks are the same as those of the third feature vectors; The third feature extraction map is formed by multiple third output picture blocks.

6. The method according to claim 2, characterized in that The fourth stage processing module includes a third fusion layer, a fourth linear embedding layer and a fourth feature network extraction layer, wherein the fourth feature network extraction layer includes a plurality of Swin Transformer networks connected in series, Wherein, the fourth stage processing module determines the target feature map in the following manner: Input the third feature extraction map into a third fusion layer to merge adjacent 2×2 third output image blocks in the third feature extraction map to obtain a plurality of fourth image blocks, wherein the image size corresponding to the fourth image blocks is a fourth preset size, and the feature dimension is a fifth feature dimension; Inputting the plurality of fourth image blocks into the fourth linear embedding layer for dimensionality reduction processing to obtain a fourth eigenvector corresponding to each fourth image block, wherein the feature dimension corresponding to the fourth eigenvector is half of the fifth feature dimension, and the image size is a fourth preset size; Inputting a plurality of fourth feature vectors into the fourth feature network extraction layer to obtain a plurality of fourth output picture blocks, wherein the picture size and feature dimension corresponding to the fourth output picture blocks are the same as those of the fourth feature vectors; The target feature map is formed by multiple fourth output image blocks.

7. The method according to claim 3, characterized in that The Swin Transformer network includes at least two normalization layers, a self-attention calculation layer and a multi-layer perceptron, wherein the normalization layer is respectively arranged before the self-attention calculation layer and the multi-layer perceptron, The Swin Transformer network obtains the first output image block in the following manner: Inputting the first feature vector into one of the standardization layers to perform standardization processing on the first feature vector to obtain a first feature vector after the standardization processing; Inputting the first feature vector after normalization into the self-attention calculation layer to perform self-attention calculation in a non-overlapping window, capture context feature information, and obtain a feature processing vector; Input the feature processing vector into another normalization layer to obtain a normalized feature processing vector; The standardized feature processing vector is input into the multi-layer perceptron for feature transformation to obtain a corresponding first output image block.

8. A 3D point cloud reconstruction device, characterized in that: Applied to a preset point cloud reconstruction model, the preset point cloud reconstruction model includes an encoder, a decoder, a context-aware fusion module and a refiner, the encoder is formed by a Swin Transformer network model, The device comprises: A feature extraction module, used to extract a target feature map corresponding to a two-dimensional input image using the encoder; A conversion module, used for performing 3D voxel conversion processing of different coarseness on the target feature map by using a decoder to obtain 3D voxels of different coarseness; A voxel fusion module, used for adaptively performing 3D reconstruction based on 3D voxels of different coarseness using a context-aware fusion module to obtain fused 3D voxels; A reconstruction module is used to optimize the fused 3D voxels through skip connections based on the refiner to generate a three-dimensional reconstruction result corresponding to the two-dimensional input image.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the 3D point cloud reconstruction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the 3D point cloud reconstruction method according to any one of claims 1 to 7 are executed.

Images