A three-dimensional model reconstruction method and device based on deep learning

Abstract

The application discloses a three-dimensional model reconstruction method and device based on deep learning, comprising the following steps: acquiring an input image; using a backbone network of an EffientNet network as a feature extractor to extract a feature map of the input image; based on the extracted feature map, inputting a trained two-dimensional detector to extract node features of multiple nodes to obtain a node encoding map, wherein the two-dimensional detector is obtained by supervising training of a first CNN network by using a pre-extracted feature map and a corresponding original image labeled; inputting the extracted node encoding map into a trained three-dimensional detector to perform three-dimensional coordinate coding on each node in a three-dimensional space to complete three-dimensional reconstruction, wherein the three-dimensional detector is obtained by supervising training of a second CNN network by using a pre-extracted node encoding map, corresponding three-dimensional coordinates and related main nodes. The method of the application can recover three-dimensional space information from a two-dimensional image and realize reconstruction of a three-dimensional model.

Description

Technical Field

[0001] This application relates to the field of AI technology, and in particular to a method and device for reconstructing 3D models based on deep learning. Background Technology

[0002] Currently, the main methods for 3D model reconstruction are as follows:

[0003] A 3D scanner is used to acquire 3D point cloud data, and then 3D reconstruction is achieved by using a convex hull algorithm and subsequent point cloud completion based on the point cloud.

[0004] Depth information is acquired using a depth camera, and positional information is extracted from 2D images. The 2D positional information and relevant depth information are combined to obtain the spatial information of the relevant model, thereby enabling the construction of a 3D model.

[0005] Spatial information is acquired using relevant motion sensors, and 3D reconstruction is achieved based on the relevant data collected by each sensor.

[0006] Existing methods for 3D reconstruction involve using 3D scanners to acquire point cloud data, followed by modeling and rendering. However, this approach is limited to specific objects; changing the object necessitates repeating the process. Using depth cameras and related sensors for 3D reconstruction is problematic due to the expensive equipment, difficult post-construction operation, negatively impacting user experience and causing inconvenience. Summary of the Invention

[0007] This application provides a method and apparatus for reconstructing a 3D model based on deep learning, which is used to recover 3D spatial information from a 2D image, thereby realizing the reconstruction of a 3D model.

[0008] This application provides a method for reconstructing a 3D model based on deep learning, including the following steps:

[0009] Obtain the input image;

[0010] The backbone of the EffientNet network is used as a feature extractor to extract feature maps from the input image;

[0011] Based on the extracted feature map, the trained two-dimensional detector is input to extract the node features of multiple nodes to obtain the node encoding map. The two-dimensional detector is obtained by supervising the training of the first CNN network using the pre-extracted feature map and the corresponding labeled original image.

[0012] The extracted node encoding map is input into the trained 3D detector to encode the 3D coordinates of each node in 3D space and complete the 3D reconstruction. The 3D detector is obtained by training a second CNN network using the pre-extracted node encoding map, the corresponding labeled 3D coordinates, and the relevant backbone nodes.

[0013] Optionally, the input image is acquired using a monocular camera.

[0014] Optionally, the backbone network of the EffientNet network includes:

[0015] The layers are arranged in sequence: a regular convolutional layer, a channel-wise convolutional layer, a cross-channel attention (SE) module layer, and a pointwise convolutional layer. The output of the pointwise convolutional layer is also used to merge the features input to the regular convolutional layer.

[0016] The kernel size of the ordinary convolutional layer is 3×3;

[0017] The kernel size of the channel-by-channel convolutional layer is 3×3;

[0018] The SE module layer is used to assign different weights to the features of different channels. It includes a first branch and a second branch. The first branch does not perform calculations and is directly multiplied with the output of the second branch. The second branch includes a global average pooling layer, a first fully connected layer, a second fully connected layer, and a sigmoid function set in sequence.

[0019] The kernel size of the pointwise convolutional layer is 1×1.

[0020] Optionally, the two-dimensional detector is trained in the following manner:

[0021] Define the position coordinates of the nodes in the given feature map as (u, v), Y p G is the pixel position of the p-th component. i The multi-class prediction function is configured to predict the position of each node in the feature map and performs the following steps:

[0022] With i=1, the response value of the multi-class prediction function is configured as follows: In the first step, the classification function G1 predicts the confidence code of the p-th component at position z.

[0023] If i > 1, input the feature confidence encoding map obtained in the previous step i-1 to calculate the multi-class prediction function. in, It is the original context map from the previous step i-1;

[0024] Iterate until the position prediction accuracy of each node meets the requirements.

[0025] Optionally, the 3D detector is trained in the following manner:

[0026] Encode the backbone nodes D in the given node encoding graph. i This forms delta maps, with each main node D... i This includes its three-dimensional coordinates, the direction vector of the main trunk where the corresponding joint is located, and the coordinates of the previous parent joint;

[0027] During training, the corresponding three-dimensional coordinates in the location map L are found based on the (u,v) coordinates corresponding to the maximum value in the node encoding map, so that the given node encoding map corresponds to the given three-dimensional coordinate points, in order to extract the location map L.

[0028] Label the extracted location map L, and repeat supervised learning until the accuracy of the obtained 3D location coordinates meets the requirements.

[0029] Optionally, the following objective loss function can be used for learning:

[0030] Joint encoding loss function in two-dimensional detectors The encoding graph H is intended to be infinitely close to the actual calibrated H. GT ,||·|| F Denotes the Frobenius norm;

[0031] Loss function of 3D delta maps D in 3D detector The three-dimensional delta maps D aim to approximate the truly calibrated G infinitely. GT , The representation is the product between matrix elements;

[0032] Loss function of location map L in 3D detector The location map L is intended to be infinitely close to the actual calibration L. GT , The representation is the product between matrix elements;

[0033] The joint length and orientation loss function L of the parent and child nodes in the main joint of the 3D detector. ref =||K GT -K||2, where K represents the changes in the length and direction of each joint;

[0034] The overall loss function is expressed as: L = L H +L delta +L Loc +L ref .

[0035] This application also proposes a computer device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the aforementioned deep learning-based 3D model reconstruction method.

[0036] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned deep learning-based 3D model reconstruction method.

[0037] The neural network structure designed in this application embodiment does not require the use of related equipment for the acquisition and calibration of a large amount of 3D data, reducing labor costs. It only requires a regular camera to achieve the conversion from 2D images to 3D object models without the need for special equipment.

[0038] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0039] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0040] Figure 1 This is a basic flowchart example of the three-dimensional model reconstruction method according to an embodiment of this application;

[0041] Figure 2 This is an example of channel-wise convolution in an embodiment of this application;

[0042] Figure 3 This is an example of the two-dimensional detector position code extraction process in an embodiment of this application;

[0043] Figure 4 This is an example of the model training process for the three-dimensional model reconstruction method in this application. Detailed Implementation

[0044] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0045] This application provides a method for reconstructing a 3D model based on deep learning, such as... Figure 1 As shown, it includes the following steps:

[0046] In step S101, an input image is acquired. Specifically, in some embodiments, the input image can be acquired by a monocular camera, such as a common commercial camera, or other devices with image acquisition capabilities. 3D reconstruction is then achieved through a subsequent network architecture. In extreme cases, the method of this application does not require additional expensive dedicated equipment.

[0047] In step S102, the backbone of the EffientNet network is used as a feature extractor to extract feature maps from the input image. In some embodiments, the backbone of the EffientNet network includes: a regular convolutional layer, a channel-wise convolutional layer, a cross-channel attention (SE) module layer, and a pointwise convolutional layer arranged sequentially. In some specific examples, the backbone of the EffientNet network is used as the feature extractor, the input 2D image size is 256×256, and the output feature map size is 32×32×256.

[0048] In step S103, based on the extracted feature map, the trained two-dimensional detector is input to extract the node features of multiple nodes to obtain a node encoding map. The two-dimensional detector is obtained by supervising the training of the first CNN network using the pre-extracted feature map and the corresponding labeled original image.

[0049] In step S104, the extracted node encoding map is input into the trained 3D detector to encode the 3D coordinates of each node in 3D space and complete the 3D reconstruction. The 3D detector is obtained by training a second CNN network using the pre-extracted node encoding map, the corresponding labeled 3D coordinates, and the relevant backbone nodes.

[0050] The neural network structure designed in this application embodiment does not require the use of related equipment for the acquisition and calibration of a large amount of 3D data, reducing labor costs. It only requires a regular camera to achieve the conversion from 2D images to 3D object models without the need for special equipment.

[0051] In some embodiments, the backbone network of EffientNet consists of five steps: ordinary convolutional layers, channel-wise convolutional layers, cross-channel attention (SE) module layers, and pointwise convolutional layers.

[0052] The kernel size of the ordinary convolutional layer is 3×3, and this layer mainly extracts preliminary image features from the input two-dimensional image.

[0053] The convolutional kernel size of the channel-wise convolutional layer is 3×3, and the number of kernels is the same as the number of channels (e.g., if there are 3 channels, the number of kernels is also 3). Each convolutional kernel extracts features for each channel individually, and the channel-wise convolution process is as follows: Figure 2 As shown.

[0054] The SE module layer assigns different weights to features in different channels, resulting in different weights in subsequent feature extraction. The SE module layer includes a first branch and a second branch. The first branch does not perform any calculations but directly multiplies with the output of the second branch. The second branch includes a globally average pooling layer, a first fully connected layer, a second fully connected layer, and a sigmoid function, arranged sequentially. Assuming the input feature dimension is c×h×w, the features in the second branch are first converted to c×1×1 dimension through globally average pooling, and then further reduced in dimension through the fully connected layers. Specifically, after passing through the first fully connected layer, the dimension is converted to... After passing through the second fully connected layer, its dimensions are restored to the previous c×1×1, and then through... The function calculates the weight value of each channel.

[0055] The pointwise convolutional layer has a 1×1 kernel size. This layer enables feature fusion and reduces feature dimensionality, thereby reducing computational cost. The output of the pointwise convolutional layer is also used to merge the features input to the ordinary convolutional layer.

[0056] In some embodiments, the two-dimensional detector is a compact, small CNN network, such as a 4-layer neural network with 3×3 convolutional kernels. The feature maps extracted in a manner similar to those described above are subjected to supervised training with the labeled real image from the original image. This allows a pixel to encode the confidence of the pixels covered by node j, thereby extracting the features of the corresponding nodes. Specifically, the two-dimensional detector is trained in the following manner:

[0057] Define the position coordinates of the nodes in the given feature map as (u, v), Y p G is the pixel position of the p-th component. i The multi-class prediction function is configured to predict the position of each node in the feature map and performs the following steps:

[0058] With i=1, the response value of the multi-class prediction function is configured as follows: It is the confidence code of the p-th component at position z predicted by the classification function G1 in the first step.

[0059] If i > 1, input the feature confidence encoding map obtained in the previous step i-1 to calculate the multi-class prediction function. in, It is the original context map from the previous step i-1.

[0060] Iterate until the position prediction accuracy of each node meets the requirements. Specifically, repeat the above steps according to actual needs, such as... Figure 3 As shown, with continuous iteration, the position prediction of each node will become more and more refined, thereby extracting the position code of the node, which is the node encoding graph.

[0061] In some embodiments, the 3D detector is used to predict the position coordinates of each node in the transformation of the reconstructed object from two-dimensional to three-dimensional space. This 3D detector is a compact, small CNN network, such as a 2-layer neural network with 3×3 convolutional kernels, using the node encoding map H extracted by the 2D detector. i With the calibrated three-dimensional coordinates and related main joints D i The direction vector is used for supervised training, thereby enabling 3D coordinate encoding of the pixels covered by each node j in 3D space. Specifically, the 3D detector is trained in the following manner:

[0062] Encode the backbone nodes D in the given node encoding graph. i This forms delta maps, with each main node D... i It includes its three-dimensional coordinates, the direction vector of the corresponding joint on the main trunk, and the coordinates of the previous parent joint. The coordinates of the parent joint are the starting coordinates of the direction vector of the corresponding joint on the main trunk.

[0063] Supervised training is performed using a 3D detector. During training, the 3D coordinates in the location map L are found based on the (u,v) coordinates corresponding to the maximum value in the node encoding map, ensuring that a given node encoding map corresponds to a given 3D coordinate point, thus extracting the location map L. In other words, during training, it is only necessary to ensure that the (u,v) coordinates corresponding to the maximum value in the node encoding map can be found in the 3D coordinates in the location map L, without considering the (u,v) coordinates of other pixels.

[0064] Label the extracted location map L, and repeat supervised learning until the accuracy of the obtained 3D location coordinates meets the requirements. For example... Figure 4As shown, the 3D model can be reconstructed based on the location map output by the 3D detector.

[0065] In some embodiments, the following objective loss function is used for learning:

[0066] Joint encoding loss function in two-dimensional detectors The encoding graph H is intended to be infinitely close to the actual calibrated H. GT ,||·|| F Denotes the Frobenius norm;

[0067] Loss function of 3D delta maps D in 3D detector The three-dimensional delta maps D aim to approximate the truly calibrated G infinitely. GT , The representation is the product between matrix elements;

[0068] Loss function of location map L in 3D detector The location map L is intended to be infinitely close to the actual calibration L. GT , The representation is the product between matrix elements;

[0069] The joint length and orientation loss function L of the parent and child nodes in the main joint of the 3D detector. ref =||K GT -K||2, where K represents the changes in the length and direction of each joint;

[0070] The overall loss function is expressed as: L = L H +L delta +L Loc +L ref .

[0071] The 3D reconstruction method of this application does not require the use of specialized equipment for the acquisition and calibration of large amounts of 3D data, thus reducing labor costs. In extreme cases, the method of this application can achieve the conversion from 2D images to 3D object models using only a common commercial camera, without the need for additional expensive specialized equipment. The method of this application enables real-time model inference and reconstruction, reducing the time cost from data acquisition to rendering and construction of the model.

[0072] This application also proposes a computer device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the aforementioned deep learning-based 3D model reconstruction method.

[0073] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned deep learning-based 3D model reconstruction method.

[0074] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0075] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0076] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0077] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A method for reconstructing a 3D model based on deep learning, characterized in that, Includes the following steps: Obtain the input image; The backbone of the EffientNet network is used as a feature extractor to extract feature maps from the input image; Based on the extracted feature map, the trained two-dimensional detector is input to extract node features of multiple key points to obtain a node encoding map. The two-dimensional detector is obtained by supervising the training of the first CNN network using the pre-extracted feature map and the corresponding labeled original image. The extracted node encoding map is input into the trained 3D detector to encode the 3D coordinates of each joint in 3D space and complete the 3D reconstruction. The 3D detector is obtained by training a second CNN network with pre-extracted node encoding map, corresponding labeled 3D coordinates and related main joints. Define the position coordinates of the joints of a given feature map as follows: , It is the first The pixel position of each component The multi-class prediction function is configured to predict the position of each node in the feature map and performs the following steps: exist In this case, the response value of the multi-class prediction function is configured as follows: , It is a classification function Predicting the first step Each component is in position z Confidence coding on; exist In the case of the previous step Input the obtained feature confidence encoding map, and calculate the multi-class prediction function. ,in, The previous step Contextual original graph mapping in; Iterate until the position prediction accuracy of each joint point meets the requirements; The 3D detector is trained in the following manner: Encode the main nodes in the given node encoding graph. ,constitute Each main joint It includes its three-dimensional coordinates, the direction vector of the main trunk where the corresponding joint is located, and the coordinates of the previous parent joint. The coordinates of the parent joint are the starting coordinates of the direction vector of the main trunk where the corresponding joint is located. Supervised training is performed using a 3D detector. During the training process, the maximum value corresponding to the node encoding graph is used. Use coordinates to find the location map The corresponding 3D coordinate values ​​are used to make the given node encoding map correspond to the given 3D coordinate points, so as to extract the location map. L， During training, it is only necessary to ensure that the maximum value in the node encoding graph corresponds to... The coordinates can be used to find the location map. The corresponding 3D coordinates in the image are not considered, excluding the coordinates of other pixels. coordinate; For the extracted location map L Label the data and repeat supervised learning until the accuracy of the obtained 3D position coordinates meets the requirements.

2. The deep learning-based 3D model reconstruction method as described in claim 1, characterized in that, The input image is acquired using a monocular camera.

3. The deep learning-based 3D model reconstruction method as described in claim 1, characterized in that, The backbone network of the EffientNet network includes: The layers are arranged in sequence: a regular convolutional layer, a channel-wise convolutional layer, a cross-channel attention (SE) module layer, and a pointwise convolutional layer. The output of the pointwise convolutional layer is also used to merge the features input to the regular convolutional layer. The kernel size of the ordinary convolutional layer is 3×3; The kernel size of the channel-by-channel convolutional layer is 3×3; The SE module layer is used to assign different weights to features of different channels. It includes a first branch and a second branch. The first branch does not perform any calculations but directly multiplies the output of the second branch. The second branch includes a globally average pooling layer, a first fully connected layer, and a second fully connected layer, set sequentially. function; The kernel size of the pointwise convolutional layer is 1×1.

4. The deep learning-based 3D model reconstruction method as described in claim 1, characterized in that, The following objective loss function is used for learning: Joint encoding loss function in two-dimensional detectors , where the coding diagram Hoping to get as close as possible to the true calibration , Denotes the Frobenius norm; 3D detector loss function , of which three-dimensional Hoping to get as close as possible to the true calibration , The representation is the product between matrix elements; Location map in 3D detector loss function Location map Hoping to get as close as possible to the true calibration , The representation is the product between matrix elements; Joint length and orientation loss function between parent and child joints in the main joint of a 3D detector ,in This represents the changes in the length and direction of each joint; The overall loss function is expressed as: .

5. A computer device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the steps of the deep learning-based 3D model reconstruction method as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the deep learning-based 3D model reconstruction method as described in any one of claims 1 to 4.

Images