Transform-based single-view and multi-view voxel three-dimensional reconstruction method

Abstract

The invention discloses a single-view and multi-view voxel three-dimensional reconstruction method based on Transform, and relates to the field of voxel three-dimensional reconstruction. The problems of'feature collapse 'in single-view reconstruction and insufficient feature relation extraction and excessive redundant features in a multi-view reconstruction task are solved. On the aspect of single-view voxel three-dimensional reconstruction, multi-scale features are extracted through Transform and CNN, global-local features are obtained through fusion of a CFV module, on the aspect of multi-view voxel three-dimensional reconstruction, similar features are aggregated through AFPM, redundancy is reduced through an FCM method based on DPC-KNN, finally, a reconstructed voxel model is obtained through a decoding layer, and the overall structure of an object is rapidly recovered.

Description

Technical Field

[0001] The invention discloses a single-view and multi-view voxel 3D reconstruction method based on Transformer, and particularly relates to the field of voxel 3D reconstruction. Background Art

[0002] Three-dimensional reconstruction based on two-dimensional images is an important research direction in computer vision and graphics. It mainly infers the 3D features of the target through its two-dimensional image, and has a wide range of application value. Traditional methods mainly rely on view geometry calculations and restore three-dimensional shapes through image shadows, textures, motion parallax and other information. The reconstruction effect depends on the accuracy and consistency of boundaries between different views. When the viewpoint difference between multiple input images is too large or the object is partially occluded, it is difficult to establish feature matching for these two-dimensional images.

[0003] With the widespread application of Transformer in the field of computer vision, it has also achieved remarkable results in the field of image-based voxel 3D reconstruction. Transformer regards the entire process as a whole, and while extracting global feature information, it solves the problem of "image order correlation" in the reconstruction process. However, in the current Transformer-based voxel 3D reconstruction method, as the number of network layers increases, the single-view Figure 3 In addition to the problem of “feature collapse” in multi-dimensional reconstruction, how to infer the relationship between the invisible and visible parts from the existing image features remains a challenge. Figure 3 In 3D reconstruction, existing models often ignore the potential connections between different views, resulting in the inability to extract key image features. As a key factor affecting the reconstruction of 3D voxel models, rich image features play a decisive role in improving the quality of model reconstruction. At the same time, too many redundant features will also affect the final reconstruction results. Therefore, how to deal with the "feature collapse" phenomenon, establish connections between different views, and minimize the interference of redundant information has become an urgent problem to be solved. Summary of the invention

[0004] The present invention discloses a single-view and multi-view voxel 3D reconstruction method based on Transformer to solve the problems in the prior art. In order to achieve the above object, the technical solution of the present invention is as follows:

[0005] A single-view voxel 3D reconstruction method based on Transformer, wherein the object reconstructed by the method is a single view, and the method comprises:

[0006] S1. Input initial information;

[0007] S2, input the initial value of the single view image into the Transformer branch and CNN group multi-scale convolution module in the network respectively, and extract the global image features and local image texture features respectively;

[0008] S3, the global-local features of the image are obtained by fusing the global features and local texture features of the image through the CFV module;

[0009] S4, inputting the global-local features of the image into a pre-designed decoder to obtain the final voxel model;

[0010] The input initial information is to use the single view image to be reconstructed as the initial value of the single view image input into the model.

[0011] The single-view image initial value is input into the CNN group multi-scale convolution module in the network to extract the local image texture features, which are obtained by the following method:

[0012] The input of the CNN group multi-scale convolution module is: in:

[0013] H represents the length of the input information; W represents the width of the input information; C represents the feature dimension to be extracted; and R represents the matrix information.

[0014] For the input image, we first use deep convolution to expand the channel dimension and get P c2f , where P c2f The channel dimension of is changed from C to 4×C.

[0015] CNN grouped multi-scale convolution module c2f The input is divided into three groups according to the channel dimension, namely g1, g2, and g3. For each group, a depth-wise separable convolutional layer with different convolution kernel sizes is introduced, specifically:

[0016] g1 obtains feature Y1 through a depth-wise separable convolution layer with a convolution kernel size of 1×1, g2 obtains feature Y2 through a depth-wise separable convolution layer with a convolution kernel size of 3×3, and g3 obtains feature Y3 through a depth-wise separable convolution layer with a convolution kernel size of 5×5.

[0017] The output Y of the module is obtained by splicing operation on the channel dimension mrfp , specifically:

[0018] Y mrfp =Concat(Y1,Y2,Y3).

[0019] Furthermore, the single-view image initial value is input into the Transformer branch in the network to extract the global image features, which are obtained by the following method:

[0020] The input to the Transformer branch is:

[0021] Where H represents the length of the input information; W represents the width of the input information; C represents the feature dimension to be extracted; and R represents the matrix information.

[0022] Introduce a residual grouped convolution module to the input image.

[0023] Among them, the residual group convolution module contains two parallel convolution branches, the upper branch includes group convolutions with convolution kernel sizes of 1×1 and 3x3, and the lower branch includes ordinary convolutions with convolution kernel sizes of 1x1 and 3x3.

[0024] Get the initial feature image X rgcb :

[0025] X rgcb =Gconv 3×3 (Gconv 1×1 (X gc ))+Conv 3×3 (Conv 1×1 (X gc ))+X gc

[0026] X gc =Gconv 1×1 (X)

[0027] Where X gc Represents a grouped convolution with a kernel size of 1×1.

[0028] The initial feature image is input into the encoding layer of Transformer to extract the global image features, which are obtained by the following method:

[0029] Divide it into N P×P patches from the length and width dimensions, where N = HW / P 2 . Reshape the dimensions of each Patch block to obtain the reshaped patches block, the patches block dimension is:

[0030] Use a linear mapping layer to flatten each patch into:

[0031] Introduce self-attention mechanism to each patch:

[0032] The three fully connected layers obtain the query key Q, key K, and value V in the self-attention mechanism:

[0033]

[0034]

[0035]

[0036] Among them, F represents the original input; B represents the new feature dimension; W k Represents the weight matrix of the bond; W q Represents the weight matrix of the query; W v A weight matrix representing the values.

[0037] The similarity between the query key Q and the key K is calculated by matrix dot multiplication to obtain the weight matrix T:

[0038] T=Q·K T .

[0039] Normalize the weight matrix T using the softmax operation:

[0040]

[0041] Where T i,j Represents the element corresponding to the i-th row and j-th column.

[0042] The encoding layer based on the Transformer module learns the features X in the image by multiplying the weight matrix T and V values att =T·O.

[0043] Using the multi-layer perception mechanism of the neural network, which consists of multiple fully connected layers and dropout layers, the final output of the Transformer encoding layer is obtained. en .

[0044] The design involves a fusion method CFV of cross-architecture global features and local texture features, which alleviates the problem of lack of internal token information interaction and non-hierarchical features in Transformer and enhances the long-range modeling ability and semantic representation ability of CNN in the following process:

[0045] The Concat operation is used to aggregate the multi-scale features Y extracted by the CNN branch in the channel dimension, where:

[0046] Y={Y1、Y2、Y3}

[0047] Global feature O integrated into the Transformer branch en , and obtain the new feature Y′, where:

[0048] Y′={Y1、Y2+O en , Y3}

[0049] Use the self-attention mechanism to get the final feature output:

[0050] O CFV =FFN(Attention(norm(Y′)))

[0051] Where norm(·) represents the normalization operation, Attention(·) represents the self-attention mechanism, and FFN(·) represents the feedforward network operation. L represents the high dimension of the input information.

[0052] The global-local features of the image are input into a pre-designed decoder to obtain the final voxel model, as follows:

[0053] In the Transformer decoder, the embedding dimension is set to 768, the number of attention heads is set to 12, and the depth is set to 8. After that, it is passed to the CNN decoder after the LayerNorm layer operation, where:

[0054] The CNN decoder consists of 3D transposed convolutions with kernel size 3×3×3 and residual blocks embedded between the 3D transposed convolution layers.

[0055] Finally, a voxel model of size 32×32×32 is output.

[0056] A multi-view voxel 3D reconstruction method based on Transformer, wherein the object reconstructed by the method is a multi-view, and the method comprises:

[0057] Input image information; use AFPM, a feature aggregation method based on attention mechanism, to achieve feature aggregation; use FCM, an image feature compression method based on DPC-KNN clustering, to reduce redundant features; use the network output layer to fuse voxels; generate the final voxel model.

[0058] The input image information is a feature sequence A of multiple images:

[0059] A={x1,x2,…x n}

[0060] in:

[0061] The feature aggregation method AFPM based on the attention mechanism is introduced to realize feature aggregation. The method adopted is as follows:

[0062] Each feature passes through a linear transformation layer without nonlinear activation function to obtain a set of learned attention scores c = {c1, c2, ..., c n}:

[0063] c n =g(xn ,W)=x n W

[0064] in g represents a linear transformation layer.

[0065] The Softmax function is introduced to standardize the learned attention scores on the input n feature sequences to obtain a new set of attention scores S n :

[0066]

[0067]

[0068] in, Represents the d-th and j-th feature sequences in N feature sequences.

[0069] Multiply the attention score S obtained above by its corresponding original feature set A to obtain the deep feature O:

[0070] O={O1,O2,…,O n}

[0071] O n =x n *S n

[0072] Among them, * represents the element-wise multiplication operation.

[0073] The weighted sum of all the obtained deep features is used to obtain the aggregated feature representation Y.

[0074] The method of introducing the image feature compression method FCM based on DPC-KNN clustering to reduce redundant features is as follows:

[0075] For the aggregate feature sequence set O = {O1, O2, ..., O n} and attention scores c = {c1,c2,…,c n},in:

[0076] O n =x n *S n , c n =g(x n ,W)=x n W, g represent linear transformation layers.

[0077] Input the set O as the query key Q in the attention mechanism into the multi-head attention, use the attention score c as the weight Weight, and the remaining feature sequences as the input of the key K and value key V in the multi-head attention:

[0078]

[0079] Among them, d k It is the dimension of Q, K, and V.

[0080] The Transformer-based 3D reconstruction voxel method further includes designing a loss function, and the loss function is Dice loss:

[0081]

[0082] Among them, p i Indicates the confidence of the predicted value at position i, gt i Represents the confidence of the true value at i. The smaller the loss function, the closer the predicted value is to the true value.

[0083] The present invention is beneficial in that:

[0084] The voxel 3D reconstruction method of the present invention does not require the use of specific technology to infer the 3D structure of an image; it does not require the camera to be calibrated in advance, and is applicable to all scenarios.

[0085] The present invention effectively solves the "feature collapse" phenomenon produced in the process of single-view voxel 3D reconstruction. Figure 3 In the multi-dimensional reconstruction, as the number of network layers increases, the features extracted in each layer tend to be homogenized. This method introduces the CNN grouped multi-scale convolution module to effectively alleviate this problem.

[0086] The present invention strengthens the potential connection between different views in the traditional Transformer, extracts key information between multiple features and uses it as a connection point, and reduces the computing power consumption caused by too many features through feature aggregation methods. Moreover, any number of image inputs can be processed in parallel, and the reconstruction result is independent of the order of image input.

[0087] The present invention can be applied to three-dimensional reconstruction voxel models of all scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 Is single vision Figure 3 The overall structure of the voxel model was reconstructed.

[0089] Figure 2 It is a schematic diagram of the residual group convolution module.

[0090] Figure 3 This is a schematic diagram of the CNN grouped multi-scale convolution module.

[0091] Figure 4 This is a schematic diagram of AFPM, a feature aggregation method based on the attention mechanism.

[0092] Figure 5 This is the structure diagram of the model decoder. DETAILED DESCRIPTION

[0093] Several embodiments of the present invention are described with reference to the accompanying drawings.

[0094] Implementation method 1, see Figures 1 to 3 This embodiment describes a single-view voxel 3D reconstruction method based on Transformer. The object reconstructed by the method is a single view. The method includes:

[0095] S1. Enter initial information.

[0096] S2. Input the initial value of the single-view image into the Transformer branch and CNN group multi-scale convolution module in the network respectively to extract global image features and local image texture features.

[0097] S3. The global-local features of the image are obtained by fusing the global features and local texture features of the image through the CFV module.

[0098] S4. The global-local features of the image are input into a pre-designed decoder to obtain the final voxel model.

[0099] The input initial information is to use the single view image to be reconstructed as the initial value of the single view image input into the model.

[0100] The single-view image initial value is input into the CNN group multi-scale convolution module in the network to extract the local image texture features, which are obtained by the following method:

[0101] The input of the CNN group multi-scale convolution module is: in:

[0102] H represents the length of the input information; W represents the width of the input information; C represents the feature dimension to be extracted; and R represents the matrix information.

[0103] For the input image, we first use deep convolution to expand the channel dimension and get P c2f , where P c2f The channel dimension of is changed from C to 4×C.

[0104] CNN group multi-scale convolution module c2f The input is divided into three groups according to the channel dimension, namely g1, g2, and g3. For each group, a depth-wise separable convolutional layer with different convolution kernel sizes is introduced, specifically:

[0105] g1 obtains feature Y1 through a depth-wise separable convolution layer with a convolution kernel size of 1×1, g2 obtains feature Y2 through a depth-wise separable convolution layer with a convolution kernel size of 3×3, and g3 obtains feature Y3 through a depth-wise separable convolution layer with a convolution kernel size of 5×5.

[0106] The output Y of the module is obtained by splicing operation on the channel dimension mrfp , specifically:

[0107] Y mrfp =Concat(Y1,Y2,Y3)

[0108] The single-view image initial value is input into the Transformer branch in the network to extract the global image features, which are obtained by the following method:

[0109] The input to the Transformer branch is: Where H represents the length of the input information; W represents the width of the input information; C represents the feature dimension to be extracted; and R represents the matrix information.

[0110] Introduce a residual grouped convolution module to the input image.

[0111] Among them, the residual group convolution module contains two parallel convolution branches, the upper branch includes group convolutions with convolution kernel sizes of 1×1 and 3x3, and the lower branch includes ordinary convolutions with convolution kernel sizes of 1x1 and 3x3.

[0112] Get the initial feature image X rgcb :

[0113] X rgcb =Gconv 3×3 (Gconv 1×1 (X gc ))+Conv 3×3 (Conv 1×1 (X gc ))+X gc

[0114] X gc =Gconv 1×1 (X)

[0115] Where X gc Represents a grouped convolution with a kernel size of 1×1.

[0116] The initial feature image is input into the encoding layer of Transformer to extract the global image features, which are obtained by the following method:

[0117] Divide it into N P×P patches from the length and width dimensions, where N = HW / P2 .

[0118] Reshape the dimensions of each patch block to obtain the reshaped patches block, the patches block dimension is:

[0119] Use a linear mapping layer to flatten each patch into:

[0120] Introduce self-attention mechanism to each patch:

[0121] The three fully connected layers obtain the query key Q, key K, and value V in the self-attention mechanism:

[0122]

[0123]

[0124]

[0125] Among them, F represents the original input; B represents the new feature dimension; W k Represents the weight matrix of the bond; W q Represents the weight matrix of the query; W v A weight matrix representing the values.

[0126] The similarity between the query key Q and the key K is calculated by matrix dot multiplication to obtain the weight matrix T:

[0127] T=Q·K T

[0128] Normalize the weight matrix T using the softmax operation:

[0129]

[0130] Where T i,j Represents the element corresponding to the i-th row and j-th column

[0131] The encoding layer based on the Transformer module learns the features X in the image by multiplying the weight matrix T and V values att :

[0132] X att =T·V

[0133] The multi-layer perception mechanism of the neural network is used, which consists of multiple fully connected layers and dropout layers. The final output of the Transformer encoding layer is obtained. en .

[0134] The design involves a fusion method CFV of cross-architecture global features and local texture features, which alleviates the problem of lack of internal token information interaction and non-hierarchical features in Transformer and enhances the long-range modeling ability and semantic representation ability of CNN in the following process:

[0135] This method uses the Concat operation to aggregate the multi-scale features Y extracted by the CNN branch in the channel dimension: Y={Y1、Y2、Y3}

[0136] Incorporating Transformer’s features O en , and obtain the new feature Y′, where:

[0137] Y′={Y1、Y2+O en , Y3}

[0138] Use the self-attention mechanism to get the final feature output:

[0139] O CFV =FFN(Attention(norm(Y′)))

[0140] Where norm(·) represents the normalization operation, Attention(·) represents the self-attention mechanism, and FFN(·) represents the feedforward network operation. Where L represents the high dimension of the input information

[0141] The global-local features of the image are input into a pre-designed decoder to obtain the final voxel model. The specific process is as follows:

[0142] In the Transformer decoder, the embedding dimension is set to 768, the number of attention heads is set to 12, and the depth is set to 8. After that, it is passed to the CNN decoder after the LayerNorm layer operation, where:

[0143] The CNN decoder consists of 3D transposed convolutions with kernel size 3×3×3 and residual blocks embedded between the 3D transposed convolution layers.

[0144] The final output is a voxel model of size 32×32×32.

[0145] The three-dimensional reconstruction voxel method also includes using the overlap IoU as an evaluation indicator:

[0146]

[0147] Where p(i,j,k) represents the predicted value of the voxel at (i,j,k), gt(i,j,k) represents the true value of the voxel at (i,j,k), t represents the voxel threshold, and I(·) represents the indicator function.

[0148] A higher IoU value represents a better reconstruction result.

[0149] Specifically, as shown in Table 1, Table 1 is a single-view Figure 3 The overlap IoU results of the three-dimensional reconstructed voxel model and other models in the test set are compared, and 43,736 three-dimensional models of 13 types provided in 3D-R2N2 are used as the data set. 32,700 of the models are divided into training sets, and the rest are divided into test sets. At the same time, 3D-R2N2 also gives 24 renderings of random perspectives for each model. These renderings are used as input to the network to show the experimental results of the three-dimensional reconstructed voxel model in a single view. Among them, the voxel threshold t is taken as 0.4, and a higher IoU value represents a better reconstruction result. It can be seen that there is a large gap in the overlap IoU of different categories. Among them, UMIFormer and the network of the present invention are both proposed based on Transformer, and the effect is significantly improved compared with the traditional 3D-R2N2 network and Pix2Vox-A network based on the CNN model.

[0150] Table 1. Comparison of the overlap IoU between the Transformer-based single-view voxel 3D reconstruction model and other models

[0151] Implementation Method 2: See Figure 4 This embodiment describes a Transformer-based multi-view voxel 3D reconstruction method, where the object reconstructed by the method is a multi-view, and the method includes:

[0152] Input image information; use AFPM, a feature aggregation method based on attention mechanism, to achieve feature aggregation; use FCM, an image feature compression method based on DPC-KNN clustering, to reduce redundant features; use the network output layer to fuse voxels; generate the final voxel model.

[0153] The input image information is a feature sequence A of multiple images:

[0154] A={x1,x2,…x n}

[0155] in:

[0156] The feature aggregation method AFPM based on the attention mechanism is introduced to realize feature aggregation. The method adopted is as follows:

[0157] Each feature passes through a linear transformation layer without nonlinear activation function to obtain a set of learned attention scores c = {c1, c2, ..., c n}:

[0158] c n =g(x n ,W)=x n W

[0159] in g represents a linear transformation layer.

[0160] The Softmax function is introduced to standardize the learned attention scores on the input n feature sequences to obtain a new set of attention scores S n

[0161]

[0162]

[0163] in, Represents the d-th and j-th feature sequences in N feature sequences.

[0164] Multiply the attention score S obtained above by its corresponding original feature set A to obtain the deep feature O:

[0165] O={O1,O2,…,O n}

[0166] O n =x n *S n

[0167] Among them, * represents the element-wise multiplication operation.

[0168] The weighted sum of all the obtained deep features is used to obtain a new feature representation Y.

[0169] The method of introducing the image feature compression method FCM based on DPC-KNN clustering to reduce redundant features is as follows:

[0170] For the aggregate feature sequence set O = {O1, O2, ..., O n} and attention scores c = {c1,c2,…,c n} as method input, where:

[0171] O n =x n *S n , c n =g(x n ,W)=x n W, g represent linear transformation layers.

[0172] Input the set O as the query key Q in the attention mechanism into the multi-head attention, use the attention score c as the weight Weight, and the remaining feature sequences as the input of the key K and value key V in the multi-head attention:

[0173]

[0174] Among them, d k It is the dimension of Q, K, and V.

[0175] Table 2 shows the comparison of the multi-view reconstruction results of this model with those of other methods. The results show that this model can reconstruct the voxel model of the object well when facing different numbers of image inputs. In addition, compared with other models in the table, this model has improved the reconstruction speed and reconstructs better results with fewer parameters.

[0176] Table 2 Comparison of IoU results between Transformer-based voxel model and other models under multi-view reconstruction

Claims

1. A single-view voxel 3D reconstruction method based on Transformer, wherein the object reconstructed by the method is a single view, and the method comprises: S1, using the single view to be reconstructed as input information of the model; S2, input the initial value of the single view image into the Transformer branch and CNN group multi-scale convolution module in the network respectively, and extract the global image features and local image texture features respectively; S3, the global-local features of the image are obtained by fusing the global features and local texture features of the image through the CFV module; S4. The global-local features of the image are input into a pre-designed decoder to obtain the final voxel model.

2. The single-view 3D reconstruction voxel method based on Transformer according to claim 1, characterized in that: The initial value of the single-view image is input into the Transformer branch and the CNN group multi-scale convolution module in the network to extract the global image features and local image texture features respectively, and the following method is used to obtain: The input of the CNN group multi-scale convolution module is: Where H represents the length of the input information; W represents the width of the input information; C represents the feature dimension to be extracted; R represents matrix information. For the input image, we first use deep convolution to expand the channel dimension and get P c2f , where P c2f The channel dimension of is changed from C to 4×C. CNN group multi-scale convolution module c2f The input is divided into three groups according to the channel dimension, namely g1, g2, and g3. For each group, a depth-wise separable convolutional layer with different convolution kernel sizes is introduced, specifically: g1 obtains feature Y1 through a depth-wise separable convolution layer with a convolution kernel size of 1×1, g2 obtains feature Y2 through a depth-wise separable convolution layer with a convolution kernel size of 3×3, and g3 obtains feature Y3 through a depth-wise separable convolution layer with a convolution kernel size of 5×5. The output Y of this module is obtained by splicing operation on the channel dimension mrfp , specifically: Y mrfp =Concat(Y1,Y2,Y3) The single-view image initialization value is input into the Transformer branch in the network to obtain the global image features, which are obtained by the following method: The input to the Transformer branch is: Where H represents the length of the input information; W represents the width of the input information; C represents the feature dimension to be extracted; R represents matrix information. Introduce a residual grouped convolution module to the input image. Among them, the residual group convolution module contains two parallel convolution branches, the upper branch includes group convolutions with convolution kernel sizes of 1×1 and 3x3, and the lower branch includes ordinary convolutions with convolution kernel sizes of 1x1 and 3x3. Get the initial feature image X rgcb : X rgcb =Gconv 3×3 (Gconv 1×1 (X gc ))+Conv 3×3 (Conv 1×1 (X gc ))+X gc X gc =Gconv 1×1 (X) Where X gc Represents a grouped convolution with a kernel size of 1×1. The initial feature image is input into the encoding layer of Transformer to extract the global image features, which are obtained by the following method: Divide it into N P×P patches from the length and width dimensions, where N = HW / P 2 . Reshape the dimensions of each patch block to obtain the reshaped patches block, the patches block dimension is: Use a linear mapping layer to flatten each patch into: Introduce self-attention mechanism to each patch: The three fully connected layers obtain the query key Q, key K, and value V in the self-attention mechanism: Among them, F represents the original input; B represents the new feature dimension; W k Represents the weight matrix of the bond; W q Represents the weight matrix of the query; W v A weight matrix representing the values. The similarity between the query key Q and the key K is calculated by matrix dot multiplication to obtain the weight matrix T: T=Q·K T Normalize the weight matrix T using the softmax operation: Where T i,j Represents the element corresponding to the i-th row and j-th column The encoding layer based on the Transformer module learns the features X in the image by multiplying the weight matrix T and V values att : X att =T·V The multi-layer perception mechanism of the neural network is used, which consists of multiple fully connected layers and dropout layers. The final output of the Transformer encoding layer is obtained. en .

3. The method for 3D reconstruction based on single view voxel according to claim 1, characterized in that: The global-local features of the image are obtained by fusing the global features and local texture features of the image through the CFV module. The specific process is as follows: Use the Concat operation to aggregate the multi-scale features Y extracted by the CNN branch in the channel dimension: Y={Y1、Y2、Y3} Incorporating Transformer’s features O en , and obtain the new feature Y′, where: <h2 style=";text-align:left;direction:ltr">Y′ = {Y1, Y2+O<h2 style=";text-align:left;direction:ltr"> en <h2 style=";text-align:left;direction:ltr"> Y3 Use the self-attention mechanism to get the final feature output: Oh CFV =FFN(Attention(norm(Y′))) Where norm(·) represents the normalization operation, Attention(·) represents the self-attention mechanism, and FFN(·) represents the feedforward network operation. Where L represents the high dimension of the input information.

4. The method for 3D reconstruction based on single view voxel according to claim 1, characterized in that: The global-local features of the image are input into a pre-designed decoder to obtain the final voxel model, as follows: In the Transformer decoder, the embedding dimension is set to 768, the number of attention heads is set to 12, and the depth is set to 8. After that, it is passed to the CNN decoder after the LayerNorm layer operation, where: The CNN decoder consists of 3D transposed convolutions with kernel size 3×3×3 and residual blocks embedded between the 3D transposed convolution layers. The final output is a voxel model of size 32×32×32.

5. A multi-view voxel 3D reconstruction method based on Transformer, wherein the object reconstructed by the method is a multi-view, and the method comprises: Input image information; use AFPM, a feature aggregation method based on attention mechanism, to achieve feature aggregation; use FCM, an image feature compression method based on DPC-KNN clustering, to reduce redundant features; use the network output layer to fuse voxels; generate the final voxel model.

6. The Transformer-based multi-view voxel 3D reconstruction method according to claim 6, characterized in that: The multiple views to be reconstructed are used as the input information of the model, and its object is a feature sequence A of multiple images, where: A={x1,x2,…x n } in:

7. The Transformer-based multi-view voxel 3D reconstruction method according to claim 6, characterized in that: The feature aggregation method AFPM based on the attention mechanism is used to realize feature aggregation, and the process is as follows: Each feature passes through a linear transformation layer without nonlinear activation function to obtain a set of learned attention scores c = {c1, c2, ..., c n }: c n =g(x n ,W)=x n W in g represents a linear transformation layer. The Softmax function is introduced to standardize the learned attention scores on the input n feature sequences to obtain a new set of attention scores S n in, Represents the d-th and j-th feature sequences in N feature sequences. Multiply the attention score S obtained above by its corresponding original feature set A to obtain the deep feature O: O={O1,O2,…,O n } O n =x n *S n Among them, * represents the element-wise multiplication operation. The weighted sum of all the obtained deep features is used to obtain the aggregated feature representation Y.

8. The Transformer-based multi-view voxel 3D reconstruction method according to claim 6, characterized in that: The image feature compression method FCM based on DPC-KNN clustering is used to reduce redundant features, and the following method is used to obtain: For the aggregate feature sequence set O = {O1, O2, ..., O n }, and attention scores c = {c1,c2,…,c n },in: O n =x n *S n , c n =g(x n ,W)=x n W, g represent linear transformation layers. Input the set O as the query key Q in the attention mechanism into the multi-head attention, use the attention score c as the weight Weight, and the remaining feature sequences as the input of the key K and value key V in the multi-head attention: Among them, d k It is the dimension of Q, K, and V.

9. The method for single-view and multi-view voxel 3D reconstruction based on Transformer according to claim 1 or claim 5, characterized in that: The Transformer-based single-view and multi-view voxel 3D reconstruction method also includes loss function design, and uses the intersection over union (IoU) as an evaluation indicator. The loss function is Dice loss: Among them, p i Indicates the confidence of the predicted value at position i, gt i Represents the confidence of the true value at i. The smaller the loss function, the closer the predicted value is to the true value. The evaluation index intersection over union (IoU) is: Where p(i,j,k) represents the predicted value of the voxel at (i,j,k), gt(i,j,k) represents the true value of the voxel at (i,j,k). t represents the voxel threshold, and I(·) represents the indicator function. A higher IoU value represents a better reconstruction result.

10. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a Transformer-based single-view and multi-view voxel three-dimensional reconstruction method as described in any one of claims 1-9.

Images