A lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation

Abstract

This invention provides a lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation, comprising the following steps: S1, constructing a student network and a teacher network; S2, both the student network and the teacher network extract 3D spatial features of the target object and perform a first progressive knowledge distillation, so that the teacher network guides the training of the student network to generate initial features of the 3D model; S3, both the student network and the teacher network perform N times of spatial feature pooling on the initial features and perform N times of second progressive knowledge distillation, so that the teacher network guides the training of the student network respectively, ultimately generating a 3D model of the target object. This method combines knowledge distillation while achieving extremely high reconstruction accuracy.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, specifically to a lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation. Background Technology

[0002] The rise of the metaverse has made 3D reconstruction a hot topic in recent years. Accurate reconstruction of 3D geometry can greatly reduce the complexity of subsequent, more detailed modeling, thanks to the remarkable learning capabilities of deep learning. Many 3D reconstruction methods have been continuously iterated and updated in recent years, resulting in increasingly higher accuracy, but at the cost of increasingly complex network models. Meanwhile, knowledge distillation has shone brightly in other fields. In image processing, knowledge distillation typically requires pre-training a teacher network with a large number of parameters and a relatively complex model on a dataset, and then using the teacher network to guide the training of a student network with fewer parameters and a simpler model. Therefore, applying knowledge distillation to 3D mesh reconstruction to reduce the complexity of the network model and the number of parameters is an obvious approach.

[0003] However, applying knowledge distillation to current 3D mesh reconstruction methods presents several challenges. Firstly, current 3D mesh reconstruction methods generally employ a progressive approach, gradually increasing the number of vertices to improve model precision. However, when applying knowledge distillation to this progressive approach, the teacher network struggles to identify the correct guidance target when guiding feature optimization because later stages have a higher vertex count than earlier stages. Secondly, if a non-progressive 3D mesh reconstruction method is adopted, either too few vertices result in low reconstruction accuracy, or too many vertices make feature optimization difficult, leading to poor reconstruction results. Summary of the Invention

[0004] Technical problems to be solved

[0005] To address the aforementioned shortcomings of existing technologies, this invention provides a lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation. This method combines knowledge distillation with extremely high reconstruction accuracy.

[0006] Technical solution

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] This invention discloses a lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation, comprising the following steps:

[0009] S1. Construct student and teacher networks;

[0010] S2. Both the student network and the teacher network extract three-dimensional spatial features of the target object and perform a first progressive knowledge distillation, so that the teacher network guides the training of the student network to generate the initial features of the three-dimensional model;

[0011] S3. Both the student network and the teacher network perform N spatial feature pooling operations on the initial features and N second progressive knowledge distillation operations, so that the teacher network guides and trains the student network respectively, and finally generates a three-dimensional model of the target object.

[0012] Furthermore, the three-dimensional spatial feature extraction specifically includes:

[0013] The spatial features of the two-dimensional image of the target object are extracted using VGG-16, and the high-level features of the two-dimensional image extracted from the 'conv3_3' layer, 'conv4_3' layer and 'conv5_3' layer are stitched together to obtain the spatial features of a two-dimensional image with misaligned image size and a total dimension of 960.

[0014] Input multiple 2D images from different viewpoints to obtain the spatial features of the 2D images from multiple views;

[0015] Spatial features of the two-dimensional images of the multiple views are extracted, and then the positions of vertices on the three-dimensional mesh model in the multiple two-dimensional images are calculated according to the camera matrix, which serves as the mapping relationship between the pixels of the two-dimensional images and the vertices of the three-dimensional mesh.

[0016] For each vertex on the 3D mesh model, calculate its 2D projection position on the high-level feature map of multiple 2D images, and use bilinear interpolation to calculate the floating-point position feature from the four pixels surrounding the floating-point position of the 2D projection position, so that each vertex on the 3D mesh model obtains multiple image features with a dimension of 960.

[0017] Calculate the maximum value, mean, and variance among the multiple image features with 960 dimensions to obtain three image features with 960 dimensions. Then, concatenate them to obtain a two-dimensional image feature with 2880 dimensions for the vertex.

[0018] Furthermore, the teacher network employs adjacency-based spatial feature pooling, while the student network employs projection-based spatial feature pooling.

[0019] Furthermore, the spatial feature pooling based on adjacency specifically includes: attaching two-dimensional image features extracted using three-dimensional spatial features to three-dimensional mesh vertices, additionally stitching three-dimensional spatial features onto the mesh vertices, and directly initializing the features of the newly added vertices using feature pooling.

[0020] Furthermore, the projection-based spatial feature pooling specifically includes: using vertex pooling to generate a fine mesh from the coarse mesh including the initial features, extracting two-dimensional image features based on the vertex coordinates of the fine mesh, and simultaneously stitching them with the three-dimensional spatial features.

[0021] Furthermore, the three-dimensional model of the target object is generated using a graph convolutional network.

[0022] Furthermore, the graph convolutional network of the teacher network consists of 14 graph convolutional layers and 192 channels in the first N-1 stages, and the 192-dimensional vertex features output by the 13th graph convolutional layer are used as the three-dimensional spatial features to be inherited in the next stage. Finally, the 14th graph convolutional layer outputs the three-dimensional spatial coordinates for each vertex. In the Nth stage, it consists of 15 graph convolutional layers and 192 channels, and the 192-dimensional three-dimensional spatial features output by the 14th graph convolutional layer are used to output the three-dimensional spatial coordinates in the 15th graph convolutional layer.

[0023] Furthermore, the graph convolutional network of the student network consists of 6 graph convolutional layers and 192 channels in the first N-1 stages, and the 192-dimensional vertex features output by the 5th graph convolutional layer are used as the 3D spatial features to be inherited in the next stage. Finally, the 6th graph convolutional layer outputs the 3D spatial coordinates for each vertex. In the Nth stage, it consists of 7 graph convolutional layers and 192 channels, and the 192-dimensional 3D spatial features output by the 6th graph convolutional layer are used to output the 3D spatial coordinates in the 7th graph convolutional layer.

[0024] Furthermore, the second progressive knowledge distillation includes dividing the three-dimensional spatial features of the teacher network into N stages to guide the spatial features of the student network, with each stage corresponding to a different feature.

[0025] Furthermore, the loss function between the teacher network and the student network is:

[0026] L all =αL1+L p2m ,

[0027] Where α is 0.75, L p2m For all loss functions in Pixel2Mesh, the coefficients remain unchanged;

[0028]

[0029] Where P represents all vertices of the prediction grid, N represents the number of channels for the vertex features, F represents the vertex features output by the teacher network, and f represents the vertex features of each layer in the student network.

[0030] Beneficial effects

[0031] This invention designs a phased, progressive 3D mesh reconstruction based on knowledge distillation, which can generate 3D geometric shapes that are close to the original model results under the condition of significantly compressing the network model structure. The method uses progressive thinking to optimize the reconstruction network and proposes a multi-stage progressive knowledge distillation. On the one hand, knowledge distillation is used in multiple stages. On the other hand, the intermediate layer features of the teacher network in multiple stages will guide the feature learning of the intermediate layers of the corresponding student networks. In addition, in order to give the newly added vertices more suitable initialization features after adding vertices, the inventors proposed a projection-based spatial feature pooling module, which allows the newly added vertices to determine the initialization features according to the projection position of the 3D spatial position on the image. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0033] Figure 1 A schematic diagram of the steps of a lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation provided in an embodiment of the present invention;

[0034] Figure 2 The overall flowchart of a lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation provided in an embodiment of the present invention is shown below.

[0035] Figure 3 A schematic diagram of the image extraction network in a lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation provided in an embodiment of the present invention;

[0036] Figure 4 (a) is a schematic diagram of the pooling process based on spatial features of adjacency;

[0037] Figure 4 (b) is a schematic diagram of the pooling process based on spatial features of projection;

[0038] Figure 5 This is a schematic diagram of three-dimensional spatial feature extraction.

[0039] Figure 6 This is a schematic diagram of vertex pooling;

[0040] Figure 7 This is a schematic diagram of a graph convolutional network based on progressive knowledge distillation. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0042] See Figure 1 and Figure 2 An embodiment of the present invention provides a lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation, comprising the following steps:

[0043] S1. Construct student and teacher networks;

[0044] S2. Both the student network and the teacher network extract three-dimensional spatial features of the target object and perform a first progressive knowledge distillation, so that the teacher network guides the training of the student network to generate the initial features of the three-dimensional model;

[0045] S3. Both the student network and the teacher network perform N spatial feature pooling operations on the initial features and N second progressive knowledge distillation operations, so that the teacher network guides and trains the student network respectively, and finally generates a three-dimensional model of the target object.

[0046] In this embodiment, refer to Figure 3 and Figure 5 The three-dimensional spatial feature extraction specifically includes:

[0047] 1) The spatial features of the two-dimensional image of the target object are extracted using VGG-16, and the high-level features of the two-dimensional image extracted by the 'conv3_3' layer, 'conv4_3' layer and 'conv5_3' layer are stitched together to obtain the spatial features of a two-dimensional image with misaligned image size and a total dimension of 960.

[0048] 2) Input multiple two-dimensional images from different viewpoints to obtain the spatial features of the two-dimensional images from multiple views;

[0049] 3) Extract spatial features from the two-dimensional images of the multiple views, and then calculate the position of the vertex on the three-dimensional mesh model in the multiple two-dimensional images based on the camera matrix, as a mapping relationship between the pixels of the two-dimensional images and the vertices of the three-dimensional mesh;

[0050] 4) For each vertex on the 3D mesh model, calculate its 2D projection position on the high-level feature map of multiple 2D images, and use bilinear interpolation to calculate the floating-point position feature from the four pixels surrounding the floating-point position of the 2D projection position, so that each vertex on the 3D mesh model obtains multiple image features with a dimension of 960.

[0051] 5) Calculate the maximum value, mean and variance among the multiple image features with 960 dimensions to obtain three image features with 960 dimensions. Then, stitch them together to obtain a two-dimensional image feature with 2880 dimensions for the vertex.

[0052] The inventors used the classic VGG-16 algorithm to extract spatial features from 2D images. They concatenated the high-level features extracted from the 'conv3_3', 'conv4_3', and 'conv5_3' layers to obtain spatial features of a 2D image with misaligned dimensions but a total dimension of 960. Since the input consists of multiple 2D images from different viewpoints, the inventors obtained spatial features from multiple views of the 2D images. 3D spatial feature extraction (…) Figure 4 The main function of this method is to extract features from a two-dimensional color image and map them onto a three-dimensional mesh model. Therefore, it first extracts features from the two-dimensional image, and then (based on the camera matrix) calculates the positions of vertices on the three-dimensional mesh model within the image, serving as a mapping from two-dimensional image pixels to three-dimensional mesh vertices, thus obtaining the features of the mesh model. For each vertex on the three-dimensional mesh model, the camera matrix is ​​used to calculate its two-dimensional projection position on high-level feature maps across multiple views. Bilinear interpolation is used to calculate the floating-point position feature from the four pixels surrounding the projected floating-point position, thus each vertex can obtain multiple image features of 960 dimensions. The inventors use statistical methods to combine the image features from multiple views. The maximum, mean, and variance among the multiple 960-dimensional image features are calculated, resulting in three 960-dimensional image features, which are then concatenated to obtain a final two-dimensional image feature of 2880 dimensions. Considering the lack of three-dimensional spatial information in the vertex features, the inventors need to inherit three-dimensional spatial features from the previous stage. For existing vertices, the 3D spatial features from the previous stage can be directly inherited. For newly added vertices, the average of the features of two adjacent vertices is calculated based on adjacency relationships to serve as the 3D spatial feature of the new vertex. The 192-dimensional 3D spatial features are concatenated with the 2880-dimensional image features and vertex coordinates, ultimately assigning each vertex a feature with a dimension of 3075. Furthermore, in the first stage, there are no inheritable 3D spatial features; therefore, only the vertex coordinates can represent the 3D spatial features. Thus, the vertex feature dimension in the first stage is 2883.

[0053] In this embodiment, the teacher network employs adjacency-based spatial feature up-pooling, and the student network employs projection-based spatial feature up-pooling. Further, the adjacency-based spatial feature up-pooling specifically includes: appending two-dimensional image features extracted using three-dimensional spatial features to the vertices of a three-dimensional mesh; additionally stitching three-dimensional spatial features onto the mesh vertices; and directly initializing the features of the newly added vertices using feature up-pooling. Further, the projection-based spatial feature up-pooling specifically includes: generating a fine mesh from the coarse mesh including the initial features using vertex up-pooling; extracting two-dimensional image features based on the vertex coordinates of the fine mesh; and simultaneously stitching them with the three-dimensional spatial features.

[0054] Feature pooling methods use the average feature of neighboring vertices as the initial features for new vertices. Vertex features come not only from the two-dimensional features of the image but also from the three-dimensional spatial features of the previous stage. Using feature pooling means that the initial features of new vertices are entirely derived from the features of neighboring vertices. This initialization method ignores the spatial position attribute of new vertices and overemphasizes the adjacency relationship of the mesh. In contrast, projection-based spatial feature pooling takes into account both the spatial position of vertices and the adjacency relationship of mesh vertices. On one hand, the inventors determine the two-dimensional image features of new vertices based on their spatial position by projecting the vertex onto the image; on the other hand, the inventors use feature pooling to obtain the three-dimensional spatial features of new vertices. This initialization method for new vertex features allows the inventors to obtain more suitable two-dimensional image features. Therefore, it can be said that projection-based spatial feature pooling allows the inventors to obtain more appropriate initial features.

[0055] Specifically, see Figure 6 The inventor added a vertex in the middle of each side and connected the newly added vertices within the same triangle, thus turning the original triangle into four new triangles. The coordinates of the newly added vertices are the average of the coordinates of two adjacent vertices in the original triangle.

[0056] Pooling modules based on the spatial characteristics of the teacher network ( Figure 4In (a)), the coarse mesh including the initial features is processed through 3D spatial feature extraction, adding 2883-dimensional image features to each vertex, and then concatenating it with 192-dimensional 3D spatial features inherited from the previous stage, ultimately adding a completely new 3075-dimensional vertex feature to each vertex of the coarse mesh including the initial features. After feature pooling, the coarse mesh including the initial features, with added vertex features, directly initializes vertex features for newly added vertices. The initial features for newly added vertices are taken from the average feature value of adjacent vertices. Since the adjacent vertex features in the teacher network simultaneously contain 2D image features and 3D spatial features, the initial feature dimension of newly added vertices is also 3075. Additionally, in the student network, the inventors used projection-based spatial feature pooling (… Figure 4 (b) After vertex pooling, the coarse mesh including the initial features only increases the mesh resolution without initializing the features of the newly added vertices. After passing through the 3D spatial feature extraction module, only 2883-dimensional 2D image features are obtained for each vertex of the fine mesh. Since the newly added vertex did not exist in the previous stage, there are no directly inheritable 3D spatial features of the fine mesh. Therefore, the 3D spatial features from the previous stage cannot be directly inherited and need to be obtained separately from the two vertices adjacent to the newly added vertex using feature pooling. The 3D spatial features of the two adjacent vertices are averaged to obtain 192-dimensional 3D spatial features of the fine mesh, which are then concatenated with the 2D image features of the fine mesh. Therefore, the initialized feature dimension obtained by this method is also 3075. This method, while not ignoring the adjacency relationship of mesh vertices, emphasizes the spatial mapping relationship between the 3D mesh vertices and the 2D image, thus making the features of the newly added vertex more consistent with the actual situation.

[0057] In this embodiment, the 3D model of the target object is generated using a graph convolutional network. Further, the graph convolutional network of the teacher network consists of 14 graph convolutional layers and 192 channels in the first N-1 stages, and the 192-dimensional vertex features output by the 13th graph convolutional layer are used as the 3D spatial features to be inherited in the next stage. Finally, the 14th graph convolutional layer outputs 3D spatial coordinates for each vertex. In the Nth stage, it consists of 15 graph convolutional layers and 192 channels, and the 192-dimensional 3D spatial features output by the 14th graph convolutional layer are used to output 3D spatial coordinates in the 15th graph convolutional layer. Furthermore, the graph convolutional network of the student network consists of 6 graph convolutional layers and 192 channels in the first N-1 stages, and the 192-dimensional vertex features output by the 5th graph convolutional layer are used as the 3D spatial features to be inherited in the next stage. Finally, the 6th graph convolutional layer outputs the 3D spatial coordinates for each vertex. In the Nth stage, it consists of 7 graph convolutional layers and 192 channels, and the 192-dimensional 3D spatial features output by the 6th graph convolutional layer are used to output the 3D spatial coordinates in the 7th graph convolutional layer.

[0058] Specifically, assuming N=3, after obtaining 3075-dimensional features for each vertex, the inventors used a very deep graph convolutional network with fast links in the teacher network. Figure 7 In the first two stages, it consists of 14 graph convolutional layers and 192 channels. The 192-dimensional vertex features output by the 13th graph convolutional layer are used as the 3D spatial features to be inherited in the next stage. Finally, the 14th graph convolutional layer outputs the 3D spatial coordinates for each vertex. In the third stage, it consists of 15 graph convolutional layers and 192 channels. The 192-dimensional 3D spatial features output by the 14th graph convolutional layer are used to output the 3D spatial coordinates in the 15th graph convolutional layer.

[0059] In the student network, the inventors used a relatively small graph convolutional network with quick links. Figure 7 In the first two stages, it consists of 6 graph convolutional layers and 192 channels. The 192-dimensional vertex features output by the 5th graph convolutional layer are used as the 3D spatial features to be inherited in the next stage. Finally, the 6th graph convolutional layer outputs the 3D spatial coordinates for each vertex. In the third stage, it consists of 7 graph convolutional layers and 192 channels. The 192-dimensional 3D spatial features output by the 6th graph convolutional layer are used to output the 3D spatial coordinates in the 7th graph convolutional layer.

[0060] In three stages, the inventors will use the three-dimensional spatial features of the teacher network to guide the learning of the student network's features in stages. Specifically, in the first stage, the 156×192 three-dimensional spatial feature output by the 13th graph convolutional layer of the teacher network is used only to guide the learning of the 156×192 three-dimensional spatial features in the student network in the same stage. In the second stage, the 618×192 three-dimensional spatial feature output by the 13th graph convolutional layer of the teacher network guides the learning of the 618×192 three-dimensional spatial features in the student network in the same stage. In the third stage, the 2466×192 three-dimensional spatial feature output by the 14th graph convolutional layer of the teacher network guides the learning of the 2466×192 three-dimensional spatial features in the student network in the same stage. Through this staged guided learning, the inventors have found suitable learning objectives for each stage of the student network.

[0061] To enable the student network to quickly learn the 3D spatial features output by the teacher network, the inventors designed a progressive knowledge distillation method. In the first and second stages, the 3D spatial features output by the teacher network guide the learning of the 3D spatial features of the 1st, 3rd, and 5th graph convolutional layers of the student network in the corresponding stages. In the third stage, the 3D spatial features output by the teacher network only guide the learning of the 3D spatial features of the 1st, 3rd, 5th, and 6th graph convolutional layers of the student network in the third stage. This progressive knowledge distillation method, which guides the learning of the 3D spatial features output by the teacher network from the bottom to the top layers of the student network, ensures that the student network aims to learn the 3D spatial features output by the teacher network from the very beginning. Furthermore, this method of guiding learning at the bottom layers of the student network can, to some extent, deepen the teacher network, while using the same features to guide learning at the higher layers ensures that the student network achieves as much of the effect as possible that of the teacher network.

[0062] A multi-stage knowledge distillation approach can address some problems encountered in knowledge distillation using progressive reconstruction methods. Firstly, in some progressive 3D shape reconstruction methods, whether increasing the number of vertices or other methods to increase the fineness of the 3D mesh, the vertices of the refined mesh after adding vertices cannot correspond one-to-one with the vertices of the coarse mesh including the initial features before adding vertices. Knowledge distillation requires the use of optimized vertex features from the teacher network to guide the learning of vertex features in the student network. Therefore, when it is impossible to align the vertices of the coarse mesh and the fine mesh including the initial features, the inventors have adopted a staged knowledge distillation method. When the student network learns vertex features at lower stages, the inventors use the corresponding, more accurate vertex features from the teacher network at those lower stages to guide the student network's learning. Similarly, when the student network learns vertex features at other stages, the inventors use the vertex features from the teacher network at the corresponding stage to guide the learning. Secondly, even if the vertices of the fine mesh can be aligned with the vertices of the coarse mesh including the initial features, it is difficult to guarantee that the aligned vertices of the fine mesh, representing the coarse mesh including the initial features, will effectively represent the rough 3D shape of the object. Therefore, it is difficult to say that the vertex features of the fine mesh are a good teacher for the coarse mesh including the initial features. However, the low-level vertex features optimized by the complex teacher network can determine and generate a relatively accurate rough 3D shape of the object, which is a suitable knowledge distillation object for the student network. Finally, if a non-progressive reconstruction method is adopted, directly using the fine mesh to reconstruct the 3D shape of the object, it means that the features of a large number of vertices need to be optimized simultaneously, and the distances that different vertices need to be displaced vary greatly. This will increase the reconstruction difficulty exponentially, and the reconstruction effect of the object will be greatly reduced. Therefore, it is difficult to reconstruct a good result after knowledge distillation.

[0063] Furthermore, compared to traditional knowledge distillation, progressive knowledge distillation uses the best-optimized features of the teacher network to guide the feature learning of each layer of the corresponding student network. This approach offers several advantages. Simply using the higher layers of the teacher network to guide the higher layers of the student network is inefficient and makes it difficult for lightweight student networks to quickly learn the optimized features of the teacher network. However, progressive learning allows the student network to be guided by the higher-level features of the teacher network at the lower layers. Compared to the unguided learning at the lower layers of the student network in traditional knowledge distillation, progressive knowledge distillation allows the student network to better learn the optimized vertex features of the teacher network. Additionally, since the lower layers of the student network are guided by the higher layers of the teacher network in progressive knowledge distillation, this structure can be approximated as a structure that deepens the neural network. Therefore, progressive knowledge distillation is a better method for knowledge distillation, both in terms of accelerating the learning efficiency of the student network and improving the feature optimization effect.

[0064] In this embodiment, the distillation loss of the teacher network and the student network is calculated by the inventors using the L1 loss between the three-dimensional spatial features of both:

[0065]

[0066] Where P represents all vertices of the prediction grid, N is the number of channels for the vertex features, F is the vertex features output by the teacher network, and f is the vertex features of each layer in the student network. The total loss function can be denoted as L. all =αL1+L p2m Where α is 0.75, L p2m It is the loss function in Pixel2Mesh, and the coefficients remain unchanged. Regarding distillation loss, the inventor only used a simple L1 loss, but L2 loss or some other loss functions could also be used.

[0067] The advantages of this invention lie in its design of a phased, progressive 3D mesh reconstruction based on knowledge distillation, which can generate 3D geometric shapes that are close to the original model results under the condition of significantly compressing the network model structure. This method uses a progressive approach to optimize the reconstruction network and proposes a multi-stage progressive knowledge distillation. On the one hand, knowledge distillation is used in multiple stages. On the other hand, the intermediate layer features of the teacher network in multiple stages will guide the feature learning of the intermediate layers of the corresponding student networks. In addition, in order to give the newly added vertices more suitable initialization features after adding vertices, the inventors proposed a projection-based spatial feature pooling module, which allows the newly added vertices to determine the initialization features based on the projection position of their 3D spatial positions on the image.

[0068] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation, characterized in that, Includes the following steps: S1. Construct student and teacher networks; S2. Both the student network and the teacher network extract three-dimensional spatial features of the target object and perform a first progressive knowledge distillation, so that the teacher network guides the training of the student network to generate the initial features of the three-dimensional model; S3. Both the student network and the teacher network perform N spatial feature pooling operations on the initial features and N second progressive knowledge distillation operations, so that the teacher network guides and trains the student network respectively, and finally generates a three-dimensional model of the target object. The teacher network uses spatial feature pooling based on adjacency relationships, and the student network uses spatial feature pooling based on projection. The spatial feature up-pooling based on adjacency specifically includes: attaching two-dimensional image features extracted using three-dimensional spatial features to the vertices of a three-dimensional mesh, additionally stitching three-dimensional spatial features onto the mesh vertices, and directly initializing the features of the newly added vertices using feature up-pooling; the spatial feature up-pooling based on projection specifically includes: using vertex up-pooling to generate a fine mesh from a coarse mesh including the initial features, extracting two-dimensional image features based on the vertex coordinates of the fine mesh, and simultaneously stitching them with the three-dimensional spatial features.

2. The lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation according to claim 1, characterized in that, The three-dimensional spatial feature extraction specifically includes: The spatial features of the two-dimensional image of the target object are extracted using VGG-16, and the high-level features of the two-dimensional image extracted from the 'conv3_3' layer, 'conv4_3' layer and 'conv5_3' layer are stitched together to obtain the spatial features of a two-dimensional image with misaligned image size and a total dimension of 960. Input multiple 2D images from different viewpoints to obtain the spatial features of the 2D images from multiple views; Spatial features of the two-dimensional images of the multiple views are extracted, and then the positions of vertices on the three-dimensional mesh model in the multiple two-dimensional images are calculated according to the camera matrix, which serves as the mapping relationship between the pixels of the two-dimensional images and the vertices of the three-dimensional mesh. For each vertex on the 3D mesh model, calculate its 2D projection position on the high-level feature map of multiple 2D images, and use bilinear interpolation to calculate the floating-point position feature from the four pixels surrounding the floating-point position of the 2D projection position, so that each vertex on the 3D mesh model obtains multiple image features with a dimension of 960. Calculate the maximum value, mean, and variance among the multiple image features with 960 dimensions to obtain three image features with 960 dimensions. Then, concatenate them to obtain a two-dimensional image feature with 2880 dimensions for the vertex.

3. The lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation according to claim 1, characterized in that, The 3D model of the target object is generated using a graph convolutional network.

4. The lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation according to claim 3, characterized in that, The teacher network's graph convolutional network consists of 14 graph convolutional layers and 192 channels in the first N-1 stages. The 192-dimensional vertex features output by the 13th graph convolutional layer are used as the 3D spatial features to be inherited in the next stage. Finally, the 14th graph convolutional layer outputs the 3D spatial coordinates for each vertex. In the Nth stage, it consists of 15 graph convolutional layers and 192 channels. The 192-dimensional 3D spatial features output by the 14th graph convolutional layer are used to output the 3D spatial coordinates in the 15th graph convolutional layer.

5. The lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation according to claim 4, characterized in that, The graph convolutional network of the student network consists of 6 graph convolutional layers and 192 channels in the first N-1 stages. The 192-dimensional vertex features output by the 5th graph convolutional layer are used as the 3D spatial features to be inherited in the next stage. Finally, the 6th graph convolutional layer outputs the 3D spatial coordinates for each vertex. In the Nth stage, it consists of 7 graph convolutional layers and 192 channels. The 192-dimensional 3D spatial features output by the 6th graph convolutional layer are used to output the 3D spatial coordinates in the 7th graph convolutional layer.

6. The lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation according to claim 5, characterized in that, The second progressive knowledge distillation involves dividing the three-dimensional spatial features of the teacher network into N stages to guide the spatial features of the student network, with each stage corresponding to a specific one.

7. The lightweight 3D reconstruction method based on multi-stage progressive knowledge distillation according to claim 1, characterized in that, The loss function between the teacher network and the student network is: ， in It is 0.

75. For all loss functions in Pixel2Mesh, the coefficients remain unchanged; The distillation loss L1 of the teacher network and the student network satisfies: in, It predicts all vertices of the grid. It is the number of channels for the vertex feature. These are the vertex features output by the teacher network. These are the vertex features of each layer in the student network.

Images