Multi-view 3D reconstruction method based on deep learning

Abstract

The present invention discloses a multi-view 3D reconstruction method based on deep learning, which comprises the following steps: step 1, multi-view image acquisition; step 2, image preprocessing; step 3, two-dimensional image feature extraction; step 4, multi-view feature fusion; step 5, combining the two-dimensional image with the three-dimensional model; step 6, optimizing the initial three-dimensional model using a cascaded deformation network, wherein the cascaded deformation network is combined with the output of the perception feature pooling layer, and after multiple deformations, the initial three-dimensional model is gradually optimized; step 7, outputting a three-dimensional mesh model, which is subjected to multiple deformations in the three-dimensional reconstruction network and finally transformed into a fine three-dimensional mesh model. The present invention belongs to the field of computer vision and graphics technology, and solves the problem in the prior art that the integrity of the three-dimensional model reconstructed using a single image from a single viewpoint is poor, the details of the three-dimensional reconstruction network are not rich enough, and the application scenarios of the three-dimensional reconstruction model are limited.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and graphics technology, and relates to a multi-view 3D reconstruction method based on deep learning. Background Art

[0002] 3D reconstruction, a hot research area in computer vision, plays a vital role in many applications, including virtual reality, autonomous driving, cultural heritage conservation, and artificial intelligence. While traditional 3D reconstruction methods have achieved success in many applications, they suffer from sensitivity to noise and occlusion, which can affect reconstruction results. In recent research and applications, deep learning technology has achieved significant progress in various 2D image processing tasks. Therefore, researchers are attempting to apply deep learning to 3D reconstruction. This has resulted in deep learning-based 3D reconstruction techniques that are generally more flexible, more robust, and better able to handle noise and occlusion.

[0003] Current deep learning-based 3D reconstruction research uses a single image from a single perspective as input, and reconstructs the 3D scene using 2D images collected from that single perspective. This results in poor integrity of the 3D model reconstructed from a single image. Therefore, there is an urgent need to develop a new method that can reconstruct a 3D model using multi-perspective image input. In other words, a multi-perspective 3D reconstruction method can combine information from multiple images to further improve the performance of the 3D reconstruction network. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-perspective 3D reconstruction method based on deep learning, which solves the problem in the prior art that the 3D model reconstructed using a single image from a single perspective has poor integrity and the details of the 3D reconstruction network are not rich enough, resulting in limited application scenarios of the 3D reconstruction model.

[0005] The technical solution adopted by the present invention is a multi-view 3D reconstruction method based on deep learning, which is implemented according to the following steps:

[0006] Step 1: multi-view image acquisition;

[0007] Step 2, image preprocessing;

[0008] Step 3, 2D image feature extraction;

[0009] Step 4: multi-view feature fusion;

[0010] Step 5, combining the two-dimensional image with the three-dimensional model;

[0011] Step 6: Optimize the initial 3D model using a cascaded deformation network. The cascaded deformation network combines the output of the perceptual feature pooling layer and gradually optimizes the initial 3D model through multiple deformations.

[0012] Step 7: Output the 3D mesh model, which is deformed multiple times in the 3D reconstruction network and finally transformed into a fine 3D mesh model.

[0013] The beneficial effects of the present invention are as follows: by collecting images of multiple perspectives, each perspective is processed by an independent image feature extraction branch; at the same time, in order to reduce the influence of irrelevant features on the reconstruction result, a pyramid feature attention network is added after the image feature extraction module, feature extraction is performed again, and the useful features of the target are selectively enhanced while suppressing irrelevant features; after that, the fusion of the image feature maps of each branch is realized by statistical features, and the fused feature vector is input to the perception feature pooling layer to realize combining the two-dimensional image with the three-dimensional reconstruction model, and then the output of the perception feature pooling layer is sent to the cascade deformation network, and the target grid model is obtained by multiple deformations. The three-dimensional reconstruction method of the present invention has better accuracy, stronger robustness and good model generalization ability, and has better practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic flow diagram of the method of the present invention;

[0015] Figure 2 It is the network structure of the two-dimensional image feature extraction module in the method of the present invention;

[0016] Figure 3 It is the pyramid feature attention network structure in the method of the present invention;

[0017] Figure 4 is a three-dimensional reconstruction network structure diagram used in the method of the present invention;

[0018] Figure 5 The pixel2mesh original image is the implementation object of Example 1 of the present invention;

[0019] Figure 6 This is the reconstruction effect of pixel2mesh in Example 1 of the present invention;

[0020] Figure 7 This is the reconstruction effect of Resnet_P2M in Example 1 of the present invention;

[0021] Figure 8 This is the reconstruction effect of Example 1 of the present invention;

[0022] Figure 9 This is the pixel2mesh original image used as the implementation target of Example 2 of the present invention;

[0023] Figure 10 This is the reconstruction effect of pixel2mesh in Example 2 of the present invention;

[0024] Figure 11 This is the reconstruction effect of Resnet_P2M in Example 2 of the present invention;

[0025] Figure 12 This is the reconstruction effect of Example 2 of the present invention;

[0026] Figure 13 This is the pixel2mesh original image used as the implementation target of Example 3 of the present invention;

[0027] Figure 14 This is the pixel2mesh reconstruction effect of Example 3 of the present invention;

[0028] Figure 15 This is the reconstruction effect of Resnet_P2M in Example 3 of the present invention;

[0029] Figure 16 This is the reconstruction effect of Example 3 of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Reference Figure 1 The multi-view 3D reconstruction method based on deep learning of the present invention is implemented according to the following steps:

[0032] Step 1: Multi-view image acquisition,

[0033] Use a calibrated digital camera to shoot the object to be reconstructed from four perspectives: front, back, left, and right. Ensure that each perspective can independently capture different sides of the object, providing clear perspective information for subsequent 3D reconstruction. The specific process is:

[0034] 1.1) When capturing images from all four perspectives, choose a time period with uniform lighting and maintain the same distance between the camera and the object to ensure consistent viewing angles.

[0035] 1.2) When shooting, the digital camera faces the front, back, left, and right sides of the target object in sequence, at a 90-degree angle to the object's surface, with the shooting axis perpendicular to each surface;

[0036] 1.3) To ensure that the images collected from the front, back, left, and right perspectives do not overlap in space, check the image from each perspective and crop the areas that overlap with other perspectives.

[0037] Step 2, image preprocessing,

[0038] The same preprocessing operations are performed on the images collected from different perspectives, including color correction and denoising. The specific process is as follows:

[0039] 2.1) Perform color correction on the image,

[0040] Using the same method to change the color values ​​of all pixels in the image, the difference between the collected image and the original image caused by ambient lighting or human factors can be reduced to a certain extent. The expression of the color-corrected image is as follows:

[0041]

[0042] Where i represents the channel in the target color space; j represents the channel in the source color space; M ij represents the conversion weight from the jth channel of the source color space to the ith channel of the target color space;

[0043] 2.2) Use median filtering to remove noise from the image and improve image quality.

[0044] This is done by moving a window across the image and replacing the center pixel with the median value of all pixels in the window.

[0045] Step 3, 2D image feature extraction,

[0046] Build an image feature extraction module. This module uses each convolutional layer to obtain feature information at different levels of the two-dimensional image. In order to reduce the impact of irrelevant features on the reconstruction results, a pyramid feature attention network (PFAN) is added after the image feature extraction module to selectively enhance the useful features of the target while suppressing irrelevant features. In addition, multiple input branches are used, and each branch independently extracts the two-dimensional image features of each viewpoint to complete the extraction of multi-view two-dimensional image features. The specific process is as follows:

[0047] 3.1) Build an image feature extraction module to extract the two-dimensional image features of each viewpoint.

[0048] Reference Figure 2 The image feature extraction module uses a 2D image feature extraction module in an end-to-end 3D reconstruction algorithm. It adopts a network structure similar to VGG16, including only the convolution layer and pooling layer in the VGG16 network. The 2D image feature extraction module has five modules, named Conv1_2, Conv2_2, Conv3_3, Conv4_3, and Conv5_3, and uses each convolution layer to obtain feature information at different levels of the 2D image.

[0049] 3.2) In order to reduce the impact of irrelevant features on the reconstruction results, a pyramid feature attention network (PFAN) is added after the image feature extraction module. The spatial attention module is used to process low-level features to suppress useless information; the pyramid feature attention network and channel attention module are used to process high-level features, thereby selectively enhancing the useful features of the target and suppressing irrelevant features such as the background.

[0050] 3.3) Using the image feature extraction module constructed above, multiple input branches are adopted, each branch independently extracts the image features of each perspective, and completes the extraction of multi-perspective two-dimensional image features.

[0051] Step 4: Multi-view feature fusion,

[0052] Calculate the statistical information (mean, maximum, standard deviation) of each view, and perform feature fusion on the outputs of multiple image feature extraction modules based on the calculation results to obtain key information in multi-view images;

[0053] For the two-dimensional image features extracted by each input branch in step 3, they need to be fused before being input into the subsequent three-dimensional reconstruction network; in the feature fusion of multi-view images, statistical features are used for fusion, and the images of each view are spliced ​​through statistical information (mean, maximum value, standard deviation) to obtain the fused image features.

[0054] Step 5: Combine the 2D image with the 3D model.

[0055] The perceptual feature pooling layer is used to combine the output of the 2D image feature extraction module with the model in the 3D space to achieve the conversion of the 2D image to the 3D model.

[0056] In the perceptual feature pooling layer, given the three-dimensional coordinates of a vertex, its two-dimensional projection on the input plane is calculated using the internal function of the digital camera. Since the vertex is a continuous position in two-dimensional space, bilinear interpolation is needed to calculate the feature corresponding to this position from the four nearby pixels.

[0057] Among them, the basic principle of bilinear interpolation is as follows:

[0058] Given four points P1(x1,y1,z1), P2(x2,y1,z2), P3(x1,y2,z3), and P4(x2 y2,z4), where (x,y) are the coordinates of the point and z is the value of the point, the value of the unknown point P(x0,y0,z0) at (x0,y0) is obtained by the following formula:

[0059]

[0060] Step 6: Optimize the initial 3D model using the cascaded deformation network.

[0061] The cascaded deformation network combines the output of the perceptual feature pooling layer and gradually optimizes the initial 3D model through multiple deformations.

[0062] Specifically, the cascaded deformation network mainly consists of a mesh deformation block and an image pooling layer. The mesh deformation block updates the position and feature vector of each vertex by combining the two-dimensional image features with the vertex coordinates in the mesh; the image pooling layer is an upsampling operation, which increases the number of vertices in the deformed mesh. The more vertices in the deformed mesh, the better the effect of the final three-dimensional mesh model.

[0063] Step 7: Output the 3D mesh model.

[0064] Since the output of the 3D reconstruction network used is a 3D mesh model (there are many ways of 3D reconstruction based on deep learning: 3D reconstruction based on voxel representation, 3D reconstruction based on point cloud representation, and 3D reconstruction based on mesh representation, etc. The output of the 3D reconstruction network constructed by the method of the present invention is a 3D reconstruction model based on mesh representation), the initial 3D model starts from a coarse mesh model, undergoes multiple deformations in the 3D reconstruction network, and is finally transformed into a fine 3D mesh model.

[0065] Example 1

[0066] The implementation object of this embodiment 1 is headphones. According to the multi-view 3D reconstruction method based on deep learning of the present invention, the following steps are included:

[0067] Step 1: Use a calibrated digital camera to photograph the headset from four perspectives: front, back, left, and right. Ensure that each perspective can independently capture different sides of the object, providing clear perspective information for subsequent 3D reconstruction. The specific process is as follows:

[0068] 1.1) When capturing images from all four perspectives, choose a time period with uniform lighting and maintain the same distance between the camera and the object to ensure consistent viewing angles.

[0069] 1.2) When photographing, the digital camera should face the front, back, left, and right sides of the headset in sequence, at a 90-degree angle to the surface, with the shooting axis perpendicular to each surface;

[0070] 1.3) To ensure that the images collected from the four perspectives do not overlap in space, check the image of each perspective and crop the areas that overlap with other perspectives.

[0071] Step 2: preprocessing the acquired multi-view images;

[0072] 2.1) Perform color correction on multi-view images. This involves using the same method to change the color values ​​of all pixels in each image, minimizing the difference between the captured image and the original image caused by ambient lighting or human factors. The basic principle is expressed as follows:

[0073]

[0074] 2.2) Use median filtering to remove noise from multi-view images. Specifically, a window is moved in the image and the central pixel is replaced by the median value of all pixels in the window.

[0075] Step 3, the two-dimensional feature extraction module in this embodiment 1 is as follows Figure 2 As shown, in order to reduce the influence of irrelevant features on the reconstruction results, an image feature extraction module is added after the image feature extraction module. Figure 3 The pyramid feature attention network shown in Figure 1 selectively enhances the useful features of the target while suppressing irrelevant features. Using the above image feature extraction module, the multi-view 2D image feature extraction is completed.

[0076] 3.1) Build an image feature extraction module. Figure 2 As shown in the figure, the image feature extraction module has five modules, named Conv1_2, Conv2_2, Conv3_3, Conv4_3, and Conv5_3. Each convolutional layer is used to obtain feature information at different levels of the two-dimensional image and extract the two-dimensional image features of each perspective;

[0077] 3.2) Introduce the pyramid feature attention network. The output features F1 and F2 of Conv1_2 and Conv2_2 are used as low-level features and sent to Figure 2 The spatial attention module in is used to suppress useless information and focus more on the target object. The output features F3, F4, and F5 of Conv3_3, Conv4_3, and Conv5_3 are used as high-level features and sent to Figure 2 The context-aware feature pyramid module and channel attention module in the model are used to obtain richer semantic information. The feature pyramid module mainly extracts features from multiple different levels and then fuses features of different scales through upsampling as its output. The channel attention module weights the features from the feature pyramid module to increase the weight of the channel containing the target information. The feature map is then resized through upsampling, and the features of the two different branches are fused as the output of the pyramid feature attention network.

[0078] 3.3) In this embodiment 1, multi-input branches are used for multi-view image feature extraction, and the image of each view enters a separate branch for two-dimensional image feature extraction.

[0079] Step 4: In this embodiment 1, the statistical information (mean, maximum, and standard deviation) is used to perform feature fusion on the outputs of the two-dimensional image feature extraction modules of each branch to obtain fused multi-view image features. This step is described in detail below:

[0080] 4.1) Neighborhood selection: Select a 32×32 pixel neighborhood at the four corners (upper left, upper right, lower left, and lower right) of each view image.

[0081] 4.2) Statistical calculation. Calculate statistics in each neighborhood, including mean (Mean), maximum (Max), and standard deviation (Std);

[0082] 4.3) Weight allocation. Set the weight vector W = [w1, w2, w3, w4], where W i represents the weight of the i-th view. In this embodiment, since the front view is more important for the 3D reconstruction task, a higher weight is given to the front view;

[0083] 4.4) Weighted statistics calculation. Weight the statistics of each perspective and calculate the weighted mean M' and maximum M' max And the standard deviation S′, the formula is as follows:

[0084]

[0085] M′ max =max(Max1,Max2,Max3,Max4),

[0086]

[0087] Among them, w i is the weight of the i-th perspective; Mean i The mean of the i-th perspective; Std i The standard deviation of the i-th viewing angle; Max i The maximum value of the i-th perspective; i represents the i-th perspective; the value of i ranges from 1 to 4.

[0088] 4.5) Similarity Calculation. Euclidean distance is used to calculate the similarity between neighborhoods from different perspectives. The smaller the distance, the more similar the neighborhoods are. The specific formula is as follows:

[0089] Assume that we want to calculate the similarity between the upper left corner neighborhood of the front view image and the back view image, and record the images of the two views as image A and image B, then:

[0090]

[0091] Among them, Distance AB is the distance between the neighborhoods of image A and image B, Mean WA is the weighted mean of image A, Mean WB is the weighted mean of image B, Std WA is the weighted standard deviation of image A, Std WB is the weighted standard deviation of image B;

[0092] Similarly, the similarity between other perspective neighborhoods can be calculated;

[0093] 4.6) Feature stitching. Based on the similarity calculation results, the view features with high similarity are stitched together;

[0094] 4.7) Input the concatenated feature vector into the subsequent network structure for further processing and analysis.

[0095] Step 5: In this embodiment 1, the fused multi-view features are input into Figure 4 The perceptual feature pooling layer shown in FIG. 4 combines the two-dimensional image with the model in the three-dimensional space according to the projection of the digital camera parameters, thereby realizing the conversion from the two-dimensional image to the three-dimensional model.

[0096] Step 6, in this embodiment 1, Figure 4 The cascaded deformation network shown partially optimizes the initial three-dimensional model. The cascaded deformation network combines the output of the perceptual feature pooling layer and gradually changes the original three-dimensional mesh model through multiple deformations.

[0097] Furthermore, the loss function plays a core role in network training. It is used to evaluate the difference between the predicted model obtained by the 3D reconstruction network and the actual model obtained using the 3D scanning device. During the training process, the parameters are adjusted according to the results of the loss function to construct the 3D mesh model. This embodiment 1 defines its loss function from aspects such as the vertices and normal vectors of the model. It consists of four parts: chamfer loss function, normal loss function, Laplacian regularization, and edge length regularization:

[0098] a) The chamfer loss function measures the distance between two sets of points. Its calculation formula is:

[0099]

[0100] Among them, P is the set of sampling points from the reconstructed mesh model, and Q is the set of sampling points in the real three-dimensional model; p and q belong to the points in the two point sets respectively.

[0101] b) The normal loss function is calculated by comparing the difference between the predicted surface normal and the true surface normal, and its calculation formula is:

[0102]

[0103] Among them, the constraint condition is k∈S, u represents a vertex in the reconstructed mesh model, S represents the set of points adjacent to the vertex u; v is the vertex closest to u in the real 3D model, which is obtained by the chamfer loss function; n represents the normal vector; n v Represents the normal vector obtained from the real 3D model.

[0104] c) Laplace regularization can effectively prevent the movement of vertices from being too large and enhance the smoothness of the model. Its calculation formula is:

[0105]

[0106] Among them, δ u ′ is the Laplace coordinate of a vertex before deformation, δ u is the Laplace coordinate of a vertex after deformation.

[0107] d) Edge length regularization constrains or optimizes the edge length of the model to improve the quality or performance of the model. Its calculation formula is:

[0108]

[0109] Step 7: Obtain the final three-dimensional mesh model of the target earphone of Example 1 through the above operations.

[0110] Below Figure 5 The image shown in FIG. 1 is used as an example for experimental description, and the reconstruction effect of the earphones by the result in step 7 is compared with that of the existing 3D reconstruction network pixel2mesh and the improved network Resnet_P2M based on pixel2mesh. Figure 5 is the original image of the headphones; Figure 6 This is the reconstruction effect of pixel2mesh; Figure 7 This is the reconstruction effect of Resnet_P2M; Figure 8 This is the reconstruction effect of the present invention. It can be seen that the method of the present invention has a better reconstruction effect on headphones.

[0111] In Example 1, the method of the present invention is compared with the existing 3D reconstruction network pixel2mesh and the improved network Resnet_P2M based on pixel2mesh using the evaluation metrics F-score and CD. The F-score is used to assess the similarity between the predicted model of the 3D reconstruction network and the actual model obtained using a 3D scanning device. A higher F-score value indicates a better reconstruction. CD is used to assess the distance between corresponding vertices in the two models. A lower CD value indicates a better reconstruction.

[0112] The calculation formula of F-score is as follows:

[0113]

[0114] Where Precision is the precision rate, which indicates the ratio of the actual number of positive samples in the predicted samples to the total number of positive samples; Recall is the recall rate, which indicates the ratio of all positive samples that are correctly classified as positive; TP is the number of samples that the model predicts as positive; FP is the number of samples that the model predicts as positive; FN is the number of samples that the model predicts as negative.

[0115] The calculation formula of CD is the calculation formula of the chamfer loss function mentioned in step 6.

[0116] Table 1 compares the experimental results of the present invention's method with those of the prior art pixel2mesh and Resnet_P2M. As can be seen from Table 1, the present invention's method achieves higher F-score values ​​and lower CD values ​​than the two prior art network results, indicating that the present invention achieves superior reconstruction of the target headphones.

[0117] Table 1. Comparison of experimental results of Example 1, pixel2mesh, and Resnet_P2M

[0118] method F-score CD Pixel2mesh 71.83 0.75 Resnet_P2M 69.82 0.87 Our invention 74.57 0.74

[0119] Example 2

[0120] According to the steps of Example 1, the object "laptop computer" is three-dimensionally reconstructed.

[0121] Below is Figures 9-12 The image shown in the figure is used as an example for experimental demonstration. The reconstruction effect of the laptop computer using the method of the present invention is obtained by following the steps of Example 1. The reconstruction effect of the method of the present invention is compared with that of the existing 3D reconstruction network pixel2mesh and the improved network Resnet_P2M based on pixel2mesh. Figure 9For laptop images; Figure 10 This is the reconstruction effect of pixel2mesh; Figure 11 This is the reconstruction effect of Resnet_P2M; Figure 12 This is the reconstruction effect of the method of the present invention. It can be seen that the method of the present invention achieves better reconstruction results for the "laptop computer." Table 2 shows the experimental comparison results of the method of the present invention with the prior art pixel2mesh and the prior art Resnet_P2M. As can be seen from Table 2, the F-score values ​​of the method of the present invention are both higher than those of the networks of the other two prior art methods, and the CD values ​​are both lower than those of the networks of the other two prior art methods, indicating that the method of the present invention achieves better reconstruction results for the laptop computer.

[0122] Table 2. Comparison of experimental results of Example 2, pixel2mesh, and Resnet_P2M

[0123] method F-score CD Pixel2mesh 80.39 0.45 Resnet_P2M 75.71 0.50 Our 81.71 0.44

[0124] Example 3

[0125] According to the steps of Example 1, the object "printer" is three-dimensionally reconstructed.

[0126] Below is Figure 13-16 The image shown in the figure is used as an example for experimental demonstration. The reconstruction effect of the printer using the method of the present invention is obtained by following the steps of Example 1. The reconstruction effect of the method of the present invention is compared with that of the existing 3D reconstruction network pixel2mesh and the improved network Resnet_P2M based on pixel2mesh. Figure 13 An image for the printer; Figure 14 This is the reconstruction effect of pixel2mesh; Figure 15 This is the reconstruction effect of Resnet_P2M; Figure 16 This is the reconstruction effect of the present invention. Figure 7 It can be seen that the method of the present invention has a better reconstruction effect on the printer.

[0127] Table 3 compares the experimental results of the present invention's method with those of the prior art pixel2mesh and Resnet_P2M. As can be seen from Table 3, the present invention's method achieves higher F-score values ​​and lower CD values ​​than the other two prior art methods, indicating that the present invention achieves better reconstruction results for printers.

[0128] Table 2. Comparison of experimental results of Example 3, pixel2mesh, and Resnet_P2M

[0129] method F-score CD Pixel2mesh 56.17 1.17 Resnet_P2M 53.87 1.21 Our invention 57.78 1.14

[0130] In summary, the deep learning multi-view 3D reconstruction method of the present invention captures images from multiple viewpoints, independently extracts image features from each viewpoint using multiple input branches, and introduces a pyramid feature attention network into the image feature extraction module to suppress the influence of irrelevant features. The multi-view image features are then fused using statistical information and fed into a subsequent perceptual feature pooling layer and a cascaded deformation network. After multiple deformations, a 3D mesh model is generated, improving the poor integrity of existing 3D reconstruction methods that use a single viewpoint.

Claims

1. A multi-view 3D reconstruction method based on deep learning, characterized by: Follow these steps to implement: Step 1: Multi-view image acquisition. Use a calibrated digital camera to shoot the object to be reconstructed from four perspectives: front, back, left, and right, to obtain clear perspective information. The specific process is: 1.1) When capturing images from all four viewing angles, choose a time period with uniform lighting and maintain the same distance between the camera and the object at each viewing angle. 1.2) When shooting, the digital camera faces the front, back, left, and right sides of the target object in sequence, at a 90-degree angle to the surface of the object, with the shooting axis perpendicular to each surface; 1.3) Ensure that the images collected from the front, back, left, and right viewpoints do not overlap spatially. Check the image from each viewpoint and crop any areas that overlap with other viewpoints. Step 2: Image preprocessing: Perform the same preprocessing operations on the collected images from different perspectives, including color correction and denoising. The specific process is as follows: 2.1) Perform color correction on the image, Using the same method to change the color values ​​of all pixels in the image, the expression of the color-corrected image is as follows: in, i represents the channels in the target color space; j represents channels in the source color space; M ij Indicates the color space from the source j channel to the target color space i The conversion weight of each channel; 2.2) Use median filtering to remove noise from the image and improve image quality. Move a window across the image and replace the center pixel with the median value of all pixels in the window; Step 3: 2D image feature extraction. The specific process is: 3.1) Build an image feature extraction module and improve the VGG16 network, retaining only the complete convolution and pooling structure from conv1_2 to conv5_3; The image feature extraction module has five modules, named Conv1_2, Conv2_2, Conv3_3, Conv4_3, and Conv5_3. Each convolutional layer obtains feature information at different levels of the two-dimensional image and extracts the two-dimensional image features of each perspective. 3.2) Introducing the pyramid feature attention network, the output features F1 and F2 of Conv1_2 and Conv2_2 are used as low-level features and fed into the spatial attention module. The output features F3, F4, and F5 of Conv3_3, Conv4_3, and Conv5_3 are used as high-level features and fed into the context-aware feature pyramid module and the channel attention module to obtain richer semantic information. Features of different scales are then fused through upsampling as the output of this module. The channel attention module weights the features from the feature pyramid module to increase the weight of the channel containing the target information. The feature map is then resized through upsampling, and the features of the two different branches are fused as the output of the pyramid feature attention network. 3.3) Using the built image feature extraction module, we use multiple input branches, each of which independently extracts image features from different viewpoints, completing the extraction of multi-view 2D image features. Step 4: Multi-view feature fusion. The 2D image feature maps extracted from each input branch in step 3 are fused and used as the input of the subsequent 3D reconstruction network. The fusion strategy guided by statistical similarity features is as follows: A 32×32 pixel neighborhood window is selected at each corner of each view image. Three channel-level statistics are calculated within each neighborhood, including the mean, maximum value, and standard deviation. Different view angles are assigned different weights, and the weighted statistics for each view angle are weighted to calculate the weighted mean, maximum value, and standard deviation. The statistical values ​​are substituted into the Euclidean distance to calculate the semantic similarity between neighborhoods of different view angles. Based on the similarity calculation results, the view features with high similarity are spliced ​​and fused. Step 5: Combine the 2D image with the 3D model. The specific process is: using the perceptual feature pooling layer to combine the output of the two-dimensional image feature extraction module with the model in the three-dimensional space to achieve the conversion of the two-dimensional image to the three-dimensional model. In the perceptual feature pooling layer, given the 3D coordinates of a vertex, its 2D projection on the input plane is calculated using the internal function of the digital camera. Since the vertex is a continuous position in 2D space, bilinear interpolation is used to calculate the feature corresponding to this position from the four nearby pixels. Step 6: Use the cascaded deformation network to optimize the initial 3D model. The cascaded deformation network combines the output of the perceptual feature pooling layer and gradually optimizes the initial 3D model after multiple deformations. Specifically, the cascaded deformation network consists of a mesh deformation block and an image pooling layer. The mesh deformation block updates the position and feature vector of each vertex by combining the two-dimensional image features with the vertex coordinates in the mesh; the image pooling layer is an upsampling operation used to increase the number of vertices in the deformed mesh. The loss function plays a core role in network training. It is used to evaluate the difference between the predicted model obtained by the 3D reconstruction network and the actual model obtained by the 3D scanning equipment. During the training process, the parameters are adjusted according to the results of the loss function to construct a 3D mesh model. The following defines its loss function from the perspective of the model's vertices and normal vectors, which consists of four parts: chamfer loss function, normal loss function, Laplace regularization, and edge length regularization: a) The chamfer loss function measures the distance between two sets of points. Its calculation formula is: ， in, is the set of sampling points from the reconstructed mesh model, is a set of sampling points in the real three-dimensional model; 、 Points belonging to two point sets respectively; b) The normal loss function is calculated by comparing the difference between the predicted surface normal and the true surface normal, and its calculation formula is: ， Among them, the constraints are k ∈ S , u Represents a vertex in the reconstructed mesh model, S Representation and Vertex u A set of adjacent points; v The distance in the real three-dimensional model u The nearest vertex, which is obtained by the chamfer loss function; n represents the normal vector; n v Represents the normal vector obtained from the real three-dimensional model; c) The calculation formula of Laplace regularization is: ， ， in, is the Laplace coordinate of a vertex before deformation, is the Laplace coordinate of a vertex after deformation; d) Edge length regularization constrains or optimizes the edge length of the model. Its calculation formula is: ； Step 7: Output the 3D mesh model, which is deformed multiple times in the 3D reconstruction network and finally transformed into a fine 3D mesh model.

Images