Multi-view three-dimensional reconstruction method based on deep learning

Abstract

The invention discloses a multi-view three-dimensional reconstruction method based on deep learning, and the method comprises the steps: 1, collecting a multi-view image; 2, preprocessing the image; step 3, two-dimensional image feature extraction; 4, performing multi-view feature fusion; 5, combining the two-dimensional image with the three-dimensional model; step 6, optimizing the initial three-dimensional model by using a cascade deformation network, and gradually optimizing the initial three-dimensional model by combining the cascade deformation network with the output of the perception feature pooling layer through multiple times of deformation; and step 7, outputting the three-dimensional grid model, and finally obtaining a fine three-dimensional grid model through multiple times of deformation in the three-dimensional reconstruction network. The invention belongs to the technical field of computer vision and graphics, and solves the problems that in the prior art, the integrity of a three-dimensional model reconstructed by using a single image of a single view angle is poor, and the details of a three-dimensional reconstruction network are not rich enough, so that the application scene of the three-dimensional reconstruction model is limited.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and graphics technology, and relates to a multi-view three-dimensional reconstruction method based on deep learning. Background Art

[0002] As a hot research direction in computer vision tasks, 3D reconstruction plays an important role in many applications such as virtual reality, autonomous driving, cultural relics protection, and artificial intelligence. Although traditional 3D reconstruction methods have been successful in many applications, their disadvantage is that they are sensitive to noise and occlusion, which affects the reconstruction results. In the latest research and applications, deep learning technology has made significant progress in various tasks of 2D image processing. Therefore, researchers have tried to apply deep learning technology to the field of 3D reconstruction, making 3D reconstruction technology based on deep learning generally more flexible, more robust, and better able to handle noise and occlusion.

[0003] In current 3D reconstruction research based on deep learning, the input is a single image from a single perspective, and the 3D scene is reconstructed using 2D images collected from a single perspective, resulting in poor integrity of the 3D model reconstructed using a single image from a single perspective. Therefore, it is urgent to develop a new method that can reconstruct a 3D model using multi-perspective image input, that is, a multi-perspective 3D reconstruction method can combine information from multiple images to further improve the performance of the 3D reconstruction network. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-perspective 3D reconstruction method based on deep learning, which solves the problem in the prior art that the integrity of the 3D model reconstructed using a single image from a single perspective is poor, the details of the 3D reconstruction network are not rich enough, and the application scenarios of the 3D reconstruction model are limited.

[0005] The technical solution adopted by the present invention is a multi-view 3D reconstruction method based on deep learning, which is implemented according to the following steps:

[0006] Step 1: multi-view image acquisition;

[0007] Step 2, image preprocessing;

[0008] Step 3, two-dimensional image feature extraction;

[0009] Step 4: multi-view feature fusion;

[0010] Step 5, combining the two-dimensional image with the three-dimensional model;

[0011] Step 6: Optimize the initial three-dimensional model using a cascaded deformation network. The cascaded deformation network combines the output of the perceptual feature pooling layer and gradually optimizes the initial three-dimensional model after multiple deformations.

[0012] Step 7: Output the 3D mesh model, which is deformed multiple times in the 3D reconstruction network and finally transformed into a fine 3D mesh model.

[0013] The beneficial effects of the present invention are that, by collecting images of multiple perspectives, each perspective is processed by an independent image feature extraction branch; at the same time, in order to reduce the influence of irrelevant features on the reconstruction result, a pyramid feature attention network is added after the image feature extraction module, and feature extraction is performed again, and the useful features of the target are selectively enhanced while suppressing irrelevant features; afterwards, the fusion of the image feature graphs of each branch is realized by statistical features, and the fused feature vector is input into the perception feature pooling layer to realize the combination of the two-dimensional image and the three-dimensional reconstruction model, and then the output of the perception feature pooling layer is sent to the cascade deformation network, and the target mesh model is obtained through multiple deformations. The three-dimensional reconstruction method of the present invention has better accuracy, stronger robustness and good model generalization ability, and has better practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic flow diagram of the method of the present invention;

[0015] Figure 2 is the network structure of the two-dimensional image feature extraction module in the method of the present invention;

[0016] Figure 3 It is the pyramid feature attention network structure in the method of the present invention;

[0017] Figure 4 is a three-dimensional reconstruction network structure diagram used in the method of the present invention;

[0018] Figure 5 is the pixel2mesh original image of the implementation object of Example 1 of the present invention;

[0019] Figure 6 This is the reconstruction effect of pixel2mesh in Example 1 of the present invention;

[0020] Figure 7 This is the reconstruction effect of Resnet_P2M in Example 1 of the present invention;

[0021] Figure 8 This is the reconstruction effect of Example 1 of the present invention;

[0022] Fig. 9 is the pixel2mesh original image of the implementation object of embodiment 2 of the present invention;

[0023] Fig.10 This is the reconstruction effect of pixel2mesh in Example 2 of the present invention;

[0024] Fig.11 This is the reconstruction effect of Resnet_P2M in Example 2 of the present invention;

[0025] Fig.12 This is the reconstruction effect of Example 2 of the present invention;

[0026] Fig.13 is the pixel2mesh original image of the implementation object of Example 3 of the present invention;

[0027] Fig.14 This is the reconstruction effect of pixel2mesh in Example 3 of the present invention;

[0028] Fig.15 This is the reconstruction effect of Resnet_P2M in Example 3 of the present invention;

[0029] Fig.16 This is the reconstruction effect of Example 3 of the present invention. DETAILED DESCRIPTION

[0030] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Reference Figure 1 The multi-view 3D reconstruction method based on deep learning of the present invention is implemented according to the following steps:

[0032] Step 1: Multi-view image acquisition.

[0033] Use a calibrated digital camera to shoot the object to be reconstructed from four perspectives: front, back, left and right, to ensure that each perspective can independently capture different sides of the object, providing clear perspective information for subsequent 3D reconstruction. The specific process is:

[0034] 1.1) When shooting images from four perspectives, choose a time period with uniform light and keep the distance between the digital camera and the object equal between different perspectives to ensure the consistency of perspective;

[0035] 1.2) When shooting, the digital camera faces the front, back, left and right sides of the target object in turn, at a 90-degree angle to the surface of the object, and the shooting axis is perpendicular to each surface;

[0036] 1.3) To ensure that the images collected from the front, back, left and right perspectives do not overlap in space, check the image from each perspective and crop the areas in the image that overlap with other perspectives.

[0037] Step 2, image preprocessing,

[0038] The same preprocessing operations are performed on the images collected from different perspectives, including color correction and denoising. The specific process is:

[0039] 2.1) Perform color correction on the image,

[0040] The same method is used to change the color values ​​of all pixels in the image, which can reduce the difference between the collected image and the original image due to ambient lighting or human factors to a certain extent. The expression of the color-corrected image is as follows:

[0041]

[0042] Where i represents the channel in the target color space; j represents the channel in the source color space; M ij represents the conversion weight from the jth channel of the source color space to the ith channel of the target color space;

[0043] 2.2) Use median filtering to remove noise from the image and improve image quality.

[0044] This is done by moving a window across the image and replacing the center pixel with the median value of all the pixels in the window.

[0045] Step 3, 2D image feature extraction,

[0046] Build an image feature extraction module, which uses each convolution layer to obtain feature information at different levels of the two-dimensional image. In order to reduce the impact of irrelevant features on the reconstruction results, a pyramid feature attention network (PFAN) is added after the image feature extraction module to selectively enhance the useful features of the target and suppress irrelevant features. In addition, multiple input branches are used, and each branch independently extracts the two-dimensional image features of each perspective to complete the extraction of multi-perspective two-dimensional image features. The specific process is:

[0047] 3.1) Build an image feature extraction module to extract the two-dimensional image features of each perspective.

[0048] Reference Figure 2 The image feature extraction module uses a 2D image feature extraction module in an end-to-end 3D reconstruction algorithm, and adopts a network structure similar to VGG16, which only includes the convolution layer and pooling layer in the VGG16 network; the 2D image feature extraction module has five modules, named Conv1_2, Conv2_2, Conv3_3, Conv4_3, and Conv5_3, and uses each convolution layer to obtain feature information at different levels of the 2D image;

[0049] 3.2) In order to reduce the impact of irrelevant features on the reconstruction results, a pyramid feature attention network (PFAN) is added after the image feature extraction module. The spatial attention module is used to process low-level features to suppress useless information; the pyramid feature attention network and channel attention module are used to process high-level features, so as to selectively enhance the useful features of the target and suppress irrelevant features such as the background.

[0050] 3.3) Using the image feature extraction module constructed above, multiple input branches are adopted, each branch independently extracts the image features of each perspective, and completes the extraction of multi-perspective two-dimensional image features.

[0051] Step 4: Multi-view feature fusion.

[0052] Calculate the statistical information (mean, maximum value, standard deviation) of each view, and fuse the outputs of multiple image feature extraction modules according to the calculation results to obtain key information in multi-view images;

[0053] For the two-dimensional image features extracted by each input branch in step 3, they need to be fused before being input into the subsequent three-dimensional reconstruction network; in the feature fusion of multi-view images, statistical features are used for fusion, and the images of each view are spliced ​​through statistical information (mean, maximum value, standard deviation) to obtain the fused image features.

[0054] Step 5: Combine the 2D image with the 3D model.

[0055] The perceptual feature pooling layer is used to combine the output of the 2D image feature extraction module with the model in the 3D space to achieve the conversion of the 2D image to the 3D model.

[0056] In the perceptual feature pooling layer, given the three-dimensional coordinates of a vertex, its two-dimensional projection on the input plane is calculated using the internal function of the digital camera. Since the vertex is a continuous position in two-dimensional space, bilinear interpolation is required to calculate the features corresponding to this position from the four nearby pixels.

[0057] Among them, the basic principle of bilinear interpolation is as follows:

[0058] Given four points P1(x1,y1,z1), P2(x2,y1,z2), P3(x1,y2,z3), P4(x2 y2,z4), where (x,y) are the coordinates of the point and z is the value of the point, the value of the unknown point P(x0,y0,z0) at (x0,y0) is obtained by the following formula:

[0059]

[0060] Step 6: Optimize the initial 3D model using the cascaded deformation network.

[0061] The cascaded deformation network combines the output of the perceptual feature pooling layer and gradually optimizes the initial 3D model after multiple deformations.

[0062] Specifically, the cascade deformation network is mainly composed of a mesh deformation block and an image pooling layer. The mesh deformation block updates the position and feature vector of each vertex by combining the two-dimensional image features with the vertex coordinates in the mesh; the image pooling layer is an upsampling operation, which increases the number of vertices in the deformed mesh. The more vertices in the deformed mesh, the better the effect of the final three-dimensional mesh model.

[0063] Step 7: Output the 3D mesh model.

[0064] Since the output of the adopted 3D reconstruction network is a 3D mesh model (there are many ways of 3D reconstruction based on deep learning: 3D reconstruction based on voxel representation, 3D reconstruction based on point cloud representation, and 3D reconstruction based on mesh representation, etc. The output of the 3D reconstruction network constructed by the method of the present invention is a 3D reconstruction model based on mesh representation), the initial 3D model is a rough mesh model, which undergoes multiple deformations in the 3D reconstruction network and is finally transformed into a fine 3D mesh model.

[0065] Example 1

[0066] The implementation object of this embodiment 1 is a headset. The implementation of the multi-view 3D reconstruction method based on deep learning of the present invention includes the following steps:

[0067] Step 1: Use a calibrated digital camera to shoot the headset from four perspectives: front, back, left, and right, to ensure that each perspective can independently capture different sides of the object, providing clear perspective information for subsequent 3D reconstruction. The specific process is as follows:

[0068] 1.1) When shooting images from four perspectives, choose a time period with uniform light and keep the distance between the digital camera and the object equal between different perspectives to ensure the consistency of perspective;

[0069] 1.2) When shooting, the digital camera faces the front, back, left and right sides of the headset in turn, at a 90-degree angle to the surface of the object, and the shooting axis is perpendicular to each surface;

[0070] 1.3) To ensure that the images collected from the four perspectives (front, back, left, and right) do not overlap in space, check the image from each perspective and crop the areas in the image that overlap with other perspectives.

[0071] Step 2, preprocessing the acquired multi-view images;

[0072] 2.1) Perform color correction on multi-view images, that is, use the same method to change the color values ​​of all pixels in each image, and minimize the difference between the collected image and the original image caused by ambient lighting or human factors. The basic principle expression is as follows:

[0073]

[0074] 2.2) Use median filtering to remove noise from multi-view images. Specifically, move a window in the image and replace the central pixel with the median of all pixels in the window.

[0075] Step 3: The two-dimensional feature extraction module in this embodiment 1 is as follows: Figure 2 As shown in the figure, in order to reduce the influence of irrelevant features on the reconstruction results, an additional Figure 3 The pyramid feature attention network shown in FIG. 4 selectively enhances the useful features of the target while suppressing irrelevant features. The above image feature extraction module is used to extract the features of multi-view two-dimensional images.

[0076] 3.1) Build an image feature extraction module. Figure 2 As shown, the image feature extraction module has five modules, named Conv1_2, Conv2_2, Conv3_3, Conv4_3, and Conv5_3, respectively. Each convolutional layer is used to obtain feature information of different levels of the two-dimensional image and extract the two-dimensional image features of each perspective;

[0077] 3.2) Introduce the pyramid feature attention network. The output features F1 and F2 of Conv1_2 and Conv2_2 are used as low-level features and sent to Figure 2 The spatial attention module in is used to suppress useless information and pay more attention to the target object. The output features F3, F4, and F5 of Conv3_3, Conv4_3, and Conv5_3 are used as high-level features and sent to Figure 2 The context-aware feature pyramid module and channel attention module in the model are used to obtain richer semantic information. The feature pyramid module mainly extracts features at multiple different levels, and then fuses features of different scales as the output of the module through upsampling. The channel attention module weights the features from the feature pyramid module to increase the weight of the channel where the target information is located. The feature map size is then adjusted through upsampling, and the features of the two different branches are fused as the output of the pyramid feature attention network.

[0078] 3.3) In this embodiment 1, multi-input branches are used for multi-view image feature extraction, and the image of each view enters a separate branch for two-dimensional image feature extraction.

[0079] Step 4, in this embodiment 1, the statistical information (mean, maximum value, standard deviation) is used to fuse the outputs of the two-dimensional image feature extraction modules of each branch to obtain the fused multi-view image features. The following is a specific description of this step:

[0080] 4.1) Neighborhood selection: Select a 32×32 pixel neighborhood at the four corners (upper left, upper right, lower left, and lower right) of each view image;

[0081] 4.2) Statistical calculation. Calculate statistics in each neighborhood, including mean, maximum value, and standard deviation.

[0082] 4.3) Weight allocation. Set the weight vector W = [w1, w2, w3, w4], where W i represents the weight of the i-th view. In this embodiment, since the front view is more important for the 3D reconstruction task, a higher weight is given to the front view;

[0083] 4.4) Weighted statistics calculation. Weight the statistics of each perspective and calculate the weighted mean M′ and maximum M′ max And standard deviation S′, the formula is as follows:

[0084]

[0085] M′ max =max(Max1,Max2,Max3,Max4),

[0086]

[0087] Among them, w i is the weight of the i-th perspective; Mean i The mean of the i-th viewing angle; Std i The standard deviation of the i-th viewing angle; Max i The maximum value of the i-th viewing angle; i represents the viewing angle; the value range of i is 1 to 4.

[0088] 4.5) Similarity calculation. The Euclidean distance is used to calculate the similarity between neighborhoods from different perspectives. The smaller the distance, the more similar the neighborhoods are. The specific formula is as follows:

[0089] Assume that we want to calculate the similarity between the upper left corner neighborhood of the front view image and the back view image, and record the images of the two views as image A and image B, then:

[0090]

[0091] Among them, Distance AB is the distance between the neighborhoods of image A and image B, Mean WA is the weighted mean of image A, Mean WB is the weighted mean of image B, Std WA is the weighted standard deviation of image A, Std WB is the weighted standard deviation of image B;

[0092] Similarly, the similarity between other perspective neighborhoods can be calculated;

[0093] 4.6) Feature stitching. Based on the results of similarity calculation, the view features with high similarity are stitched together;

[0094] 4.7) Input the concatenated feature vector into the subsequent network structure for further processing and analysis.

[0095] Step 5: In this embodiment 1, the fused multi-view features are input into Figure 4 The perceptual feature pooling layer shown in the figure combines the two-dimensional image with the model in the three-dimensional space according to the projection of the digital camera parameters, thereby realizing the conversion of the two-dimensional image to the three-dimensional model.

[0096] Step 6, in this embodiment 1, use Figure 4 The cascaded deformation network shown partially optimizes the initial three-dimensional model. The cascaded deformation network combines the output of the perceptual feature pooling layer and gradually changes the original three-dimensional mesh model through multiple deformations.

[0097] Furthermore, the loss function plays a core role in network training, and is used to evaluate the difference between the prediction model obtained by the 3D reconstruction network and the actual model obtained by the 3D scanning device. During the training process, the parameters are adjusted according to the results of the loss function, and then the 3D mesh model is constructed. This embodiment 1 defines its loss function from aspects such as the vertices and normal vectors of the model, and consists of four parts: Chamfer loss function, normal loss function, Laplacian regularization, and edge length regularization:

[0098] a) The chamfer loss function measures the distance between two sets of points, and its calculation formula is:

[0099]

[0100] Among them, P is the set of sampling points from the reconstructed mesh model, Q is the set of sampling points in the real three-dimensional model; p and q belong to the points in the two point sets respectively.

[0101] b) The normal loss function is calculated by comparing the difference between the predicted surface normal and the true surface normal, and its calculation formula is:

[0102]

[0103] Among them, the constraint condition is k∈S, u represents a vertex in the reconstructed mesh model, S represents the set of points adjacent to vertex u; v is the vertex closest to u in the real 3D model, which is obtained by the chamfer loss function; n represents the normal vector; n v Represents the normal vector obtained from the real 3D model.

[0104] c) Laplace regularization can effectively prevent the movement of vertices from being too large and enhance the smoothness of the model. Its calculation formula is:

[0105]

[0106] Among them, δ u ′ is the Laplace coordinate of a vertex before deformation, δ u is the Laplace coordinate of a vertex after deformation.

[0107] d) Edge length regularization constrains or optimizes the edge length of the model to improve the quality or performance of the model. The calculation formula is:

[0108]

[0109] Step 7: The final three-dimensional mesh model of the earphones used as the implementation object of the first embodiment is obtained through the above operations.

[0110] Below Figure 5 The image shown in the figure is used as an example for experimental explanation, and the reconstruction effect of the earphone by the result in step 7 is compared with the existing 3D reconstruction network pixel2mesh and the improved network Resnet_P2M based on pixel2mesh. Figure 5 is the original image of the headphones; Figure 6 This is the reconstruction effect of pixel2mesh; Figure 7 This is the reconstruction effect of Resnet_P2M; Figure 8 This is the reconstruction effect of the present invention. It can be seen that the method of the present invention has a better reconstruction effect on the earphone.

[0111] At the same time, this embodiment 1 uses the evaluation index F-score and CD index to compare the method of the present invention with the existing 3D reconstruction network pixel2mesh and the improved network Resnet_P2M based on pixel2mesh. F-score is used to evaluate the similarity between the prediction model of the 3D reconstruction network and the actual model obtained by the 3D scanning device. The larger the F-score value, the better the reconstruction effect. CD is used to evaluate the distance between each corresponding vertex in the two models. The smaller the CD value, the better the reconstruction effect.

[0112] The calculation formula of F-score is as follows:

[0113]

[0114] Where Precision is the precision rate, which indicates the ratio of the actual number of positive samples in the predicted samples to the total number of positive samples; Recall is the recall rate, which indicates the ratio of all positive samples that are correctly classified as positive; TP indicates the number of samples that the model predicts as positive; FP indicates the number of samples that the model predicts as positive; and FN indicates the number of samples that the model predicts as negative.

[0115] The calculation formula of CD is the calculation formula of the chamfer loss function mentioned in step 6.

[0116] Table 1 shows the experimental results comparison of the method of the present invention and the prior art pixel2mesh and the prior art Resnet_P2M. It can be seen from Table 1 that the F-score values ​​of the method of the present invention are greater than the network results of the other two prior arts, and the CD values ​​are less than the network results of the other two prior arts, that is, the method of the present invention has a better reconstruction effect on the object earphone.

[0117] Table 1. Comparison of experimental results of Example 1, pixel2mesh and Resnet_P2M

[0118] method F-score CD Pixel2mesh 71.83 0.75 Resnet_P2M 69.82 0.87 Our(Invention) 74.57 0.74

[0119] Example 2

[0120] According to the steps of Example 1, the object "laptop computer" is three-dimensionally reconstructed.

[0121] Below Figure 9-12 The image shown in the figure is used as an example for experimental description, and the reconstruction effect of the method of the present invention on the laptop is obtained according to the steps of Example 1. The reconstruction effect of the method of the present invention is compared with the reconstruction effect of the laptop by the existing three-dimensional reconstruction network pixel2mesh and the improved network Resnet_P2M based on pixel2mesh. Fig. 9For laptop images; Fig.10 This is the reconstruction effect of pixel2mesh; Fig.11 This is the reconstruction effect of Resnet_P2M; Fig.12 is the reconstruction effect of the method of the present invention. It can be seen that the method of the present invention has a better reconstruction effect on the "laptop computer". Table 2 shows the experimental results of the method of the present invention and the pixel2mesh of the prior art and the Resnet_P2M of the prior art. It can be seen from Table 2 that the F-score values ​​of the present invention are greater than the network results of the other two prior arts, and the CD values ​​are less than the network results of the other two prior arts, that is, the method of the present invention has a better reconstruction effect on the laptop computer.

[0122] Table 2. Comparison of experimental results of Example 2, pixel2mesh and Resnet_P2M

[0123] method F-score CD Pixel2mesh 80.39 0.45 Resnet_P2M 75.71 0.50 Our(Invention) 81.71 0.44

[0124] Example 3

[0125] According to the steps of Example 1, the object "printer" is three-dimensionally reconstructed.

[0126] Below Figure 13-Figure 16 The image shown in the figure is used as an example for experimental description, and the reconstruction effect of the method of the present invention on the printer is obtained according to the steps of Example 1. The reconstruction effect of the method of the present invention is compared with the reconstruction effect of the existing three-dimensional reconstruction network pixel2mesh and the improved network Resnet_P2M based on pixel2mesh on the printer. Fig.13 An image for a printer; Fig.14 This is the reconstruction effect of pixel2mesh; Fig.15 This is the reconstruction effect of Resnet_P2M; Fig.16 This is the reconstruction effect of the present invention. Figure 7 It can be seen that the method of the present invention has a better reconstruction effect on the printer.

[0127] Table 3 shows the experimental results of the method of the present invention, pixel2mesh of the prior art, and Resnet_P2M of the prior art. It can be seen from Table 3 that the F-score value of the method of the present invention is greater than the network results of the other two prior arts, and the CD value is less than the network results of the other two prior arts, that is, the method of the present invention has a better reconstruction effect on the printer.

[0128] Table 2. Comparison of experimental results of Example 3, pixel2mesh and Resnet_P2M

[0129] method F-score CD Pixel2mesh 56.17 1.17 Resnet_P2M 53.87 1.21 Our(Invention) 57.78 1.14

[0130] In summary, the multi-view 3D reconstruction method of deep learning of the present invention takes images from multiple viewpoints, uses multi-input branches to independently extract image features from each viewpoint, and introduces a pyramid feature attention network in the image feature extraction module to suppress the influence of irrelevant features. Then, the multi-view image features are fused using statistical information, and sent to the subsequent perceptual feature pooling layer and cascade deformation network. After multiple deformations, a 3D mesh model is obtained, which improves the problem of poor integrity when reconstructing using a single viewpoint in the existing 3D reconstruction method.

Claims

1. A multi-view 3D reconstruction method based on deep learning, characterized in that: Follow these steps to implement: Step 1: multi-view image acquisition; Step 2, image preprocessing; Step 3, two-dimensional image feature extraction; Step 4: multi-view feature fusion; Step 5, combining the two-dimensional image with the three-dimensional model; Step 6: Optimize the initial three-dimensional model using a cascaded deformation network. The cascaded deformation network combines the output of the perceptual feature pooling layer and gradually optimizes the initial three-dimensional model after multiple deformations. Step 7: Output the 3D mesh model, which is deformed multiple times in the 3D reconstruction network and finally transformed into a fine 3D mesh model.

2. The multi-view 3D reconstruction method based on deep learning according to claim 1, characterized in that: In step 1, a calibrated digital camera is used to shoot the object to be reconstructed from four perspectives: front, back, left, and right, to obtain clear perspective information. The specific process is: 1.1) When taking images from four viewing angles, choose a time period with uniform light and keep the distance between the digital camera and the object equal between different viewing angles; 1.2) When shooting, the digital camera faces the front, back, left and right sides of the target object in turn, at a 90-degree angle to the surface of the object, and the shooting axis is perpendicular to each surface; 1.3) Ensure that the images collected from the front, back, left, and right perspectives do not overlap in space. Check the image from each perspective and crop the areas in the image that overlap with other perspectives.

3. The multi-view 3D reconstruction method based on deep learning according to claim 1, characterized in that: In step 2, the same preprocessing operation is performed on the collected images of different perspectives, including color correction and denoising. The specific process is: 2.1) Perform color correction on the image, Using the same method to change the color values ​​of all pixels in the image, the expression of the color corrected image is as follows: Where i represents the channel in the target color space; j represents the channel in the source color space; M ij represents the conversion weight from the jth channel of the source color space to the ith channel of the target color space; 2.2) Use median filtering to remove noise from the image and improve image quality. Move a window across the image and replace the center pixel with the median value of all pixels in the window.

4. The multi-view 3D reconstruction method based on deep learning according to claim 1, characterized in that: In step 3, the specific process is: 3.1) Build an image feature extraction module to extract the two-dimensional image features of each perspective. The image feature extraction module uses a 2D image feature extraction module in an end-to-end 3D reconstruction algorithm, which only includes the convolution layer and pooling layer in the VGG16 network; the 2D image feature extraction module has five modules, named Conv1_2, Conv2_2, Conv3_3, Conv4_3, and Conv5_3, and uses each convolution layer to obtain feature information at different levels of the 2D image; 3.2) Add a pyramid feature attention network after the image feature extraction module, use the spatial attention module to process low-level features; use the pyramid feature attention network and channel attention module to process high-level features, selectively enhance the useful features of the target, and suppress irrelevant features such as the background; 3.3) Using the image feature extraction module constructed above, multiple input branches are adopted, each branch independently extracts the image features of each perspective, and completes the extraction of multi-perspective two-dimensional image features.

5. The multi-view 3D reconstruction method based on deep learning according to claim 1, characterized in that: In step 4, for the two-dimensional image features extracted by each input branch in step 3, they need to be fused before being input into the subsequent three-dimensional reconstruction network; statistical features are used for fusion in the feature fusion of multi-view images, and the images of each view are spliced ​​through statistical information to obtain the fused image features.

6. The multi-view 3D reconstruction method based on deep learning according to claim 1, characterized in that: In step 5, the specific process is: using the perceptual feature pooling layer to combine the output of the 2D image feature extraction module with the model in the 3D space to achieve the conversion of the 2D image to the 3D model. In the perceptual feature pooling layer, given the three-dimensional coordinates of a vertex, its two-dimensional projection on the input plane is calculated using the internal function of the digital camera. Since the vertex is a continuous position in two-dimensional space, bilinear interpolation is used to calculate the features corresponding to this position from the four nearby pixels.

7. The multi-view 3D reconstruction method based on deep learning according to claim 1, characterized in that: In step 6, specifically, the cascade deformation network consists of a mesh deformation block and an image pooling layer. The mesh deformation block updates the position and feature vector of each vertex by combining the two-dimensional image features with the vertex coordinates in the mesh; the image pooling layer is an upsampling operation used to increase the number of vertices of the deformed mesh.

Images