Three-dimensional reconstruction method, system, machine device and computer-readable storage medium

Abstract

The present invention discloses a three-dimensional reconstruction method, system, machine device, and storage medium. The method includes: extracting features from a given image through each layer of a deep network, mapping feedforward network information from an integrated merging layer to a feedback network to perform feature extraction at a convolutional layer to obtain two-dimensional features; performing sparse cost control on the two-dimensional features, transforming the two-dimensional features into three-dimensional features, and obtaining cost information corresponding to the three-dimensional features; extracting context information through mapping the cost information to generate an initial depth map; propagating and optimizing the initial depth map to output a high-precision depth map; generating a three-dimensional point cloud from the high-precision depth map, and constructing a three-dimensional model of the given image from the three-dimensional point cloud. In implementing this three-dimensional reconstruction, memory consumption is reduced, and under sparse cost control and mapping of cost information, the amount of computation is reduced to ensure the three-dimensional reconstruction effect, and the speed requirements of the three-dimensional reconstruction can be met.

Description

Technical Field

[0001] The present invention relates to the field of computer vision application technology, and in particular to a three-dimensional reconstruction method, system, machine equipment and computer-readable storage medium. Background Art

[0002] 3D reconstruction technology has long been a key area of ​​computer vision research. 3D reconstruction is used to create computer-processable 3D digital models of 3D objects. 3D reconstruction can be achieved primarily through geometric modeling and the use of 3D scanners. Geometric modeling is the most mature technology, enabling precise 3D reconstruction. However, it suffers from complex and time-consuming 3D reconstruction processes, and poor reconstruction results for complex objects or human models.

[0003] Secondly, 3D reconstruction using 3D scanners uses devices such as structured light, depth cameras, and laser scanners to generate 3D models. However, these devices are expensive and require stringent operating conditions, making them impractical for use in less-developed facilities.

[0004] The implementation process of three-dimensional reconstruction is highly complex due to geometric model building, and three-dimensional scanners rely on expensive equipment and operate in harsh environments, which also makes the implementation of three-dimensional reconstruction highly complex. This will limit the performance of three-dimensional reconstruction technology and fail to meet real-time requirements. Summary of the Invention

[0005] In order to solve the technical problem in the related art that the high complexity and time length of three-dimensional reconstruction cannot be controlled, the present invention provides a three-dimensional reconstruction method, system, machine equipment and computer-readable storage medium.

[0006] A three-dimensional reconstruction method, comprising:

[0007] In the feature extraction of a given image through each layer of the deep network, the feedforward network information performed by the integrated merging layer is mapped to the feedback network to perform feature extraction of the convolution layer to obtain the two-dimensional features of the given image;

[0008] Performing sparse cost control on the two-dimensional features and transforming the two-dimensional features into three-dimensional features to obtain cost information corresponding to the three-dimensional features;

[0009] Extracting context information by mapping the cost information corresponding to the three-dimensional features to generate an initial depth map predicted for the given image;

[0010] Propagating and optimizing the initial depth map to output a high-precision depth map;

[0011] A three-dimensional point cloud is generated for the high-precision depth map, and a three-dimensional model of the given image is constructed based on the three-dimensional point cloud.

[0012] In an exemplary embodiment, the combination of the merging layer and the convolution layer constitutes a network layer for performing feature extraction in the deep network, and the feature extraction of a given image in the deep network through each network layer is performed by mapping the feedforward network information performed by the integrated merging layer to the feedback network to perform feature extraction of the convolution layer, thereby obtaining a two-dimensional feature of the given image, including:

[0013] In the feature extraction of each layer of the deep network, for the back propagation from the feedforward network to the feedback network in the merging layer, the information stored in the feedforward network is reversed and forward propagated to recover the information required for the back propagation;

[0014] The convolution features extracted by the convolution layer for a given image are input into a feedback network for recovering the information required for back propagation, and the feedback network outputs the two-dimensional features corresponding to the convolution layer.

[0015] In an exemplary embodiment, in the feature extraction of each network layer in the deep network, for the back propagation from the feedforward network to the feedback network in the merging layer, the information stored in the feedforward network is flipped to the forward propagation calculation to restore the information required for the back propagation, including:

[0016] Extracting features of the given image in the deep network, forward propagating through the feedforward network in the merging layer and then inputting them into the convolutional layer, where the convolutional layer performs feature extraction;

[0017] The information obtained by the forward propagation process and stored by the feedforward network is subjected to the forward propagation flip calculation, and the information required for the reverse propagation is recovered through the flip calculation.

[0018] In an exemplary embodiment, performing sparse cost control on the two-dimensional features and transforming the two-dimensional features into three-dimensional features to obtain cost information corresponding to the three-dimensional features includes:

[0019] Generating a cost amount by cost amount measurement on the two-dimensional feature and performing a sparse operation to obtain a sparse cost amount, wherein the sparse cost amount is used to perform capacity control on context information extraction of the given image;

[0020] The two-dimensional features are converted into three-dimensional features through the sparse cost amount to obtain cost amount information corresponding to the three-dimensional features.

[0021] In an exemplary embodiment, extracting context information by mapping the cost information corresponding to the three-dimensional features to generate an initial depth map predicted for the given image includes:

[0022] In each layer of the network constructed for context information extraction, the input cost information is mapped through the convolution gate set in the network layer to extract the context information and generate an initial depth map predicted for the given image.

[0023] In an exemplary embodiment, the convolution gate is a first convolution gate, and in each layer of the network constructed for context information extraction, mapping the input cost information through the convolution gate set in the network layer to extract the context information to generate an initial depth map predicted for the given image includes:

[0024] Mapping the cost information corresponding to the three-dimensional feature to the visual space using the configured convolutional layer as the first convolution gate;

[0025] The mapping of the cost information in the visual space is performed, and context information is extracted by coordinating the influence control of the previous step with the reset gate, the update gate, and the second convolution gate configured by the update gate to generate an initial depth map predicted for the given image.

[0026] In an exemplary embodiment, the propagating and optimizing the initial depth map to output a high-precision depth map includes:

[0027] Performing a coarse optimization operation on the initial depth map by propagating the initial depth map through a convolutional network;

[0028] The output of the coarse optimization operation is used as an input object, and the initial depth map is refined to generate a high-precision depth map.

[0029] A three-dimensional reconstruction system, comprising:

[0030] A feature construction module is used to extract features of a given image in the deep network through each layer of the network, by mapping the feedforward network information performed by the integrated merging layer to the feedback network to perform convolutional layer feature extraction, thereby obtaining a two-dimensional feature of the given image;

[0031] a sparse cost control module, configured to perform sparse cost control on the two-dimensional features, and transform the two-dimensional features into three-dimensional features to obtain cost information corresponding to the three-dimensional features;

[0032] A depth prediction module, configured to extract context information by mapping the cost information corresponding to the three-dimensional features, and generate an initial depth map predicted for the given image;

[0033] A precision optimization module, configured to propagate and optimize the initial depth map and output a high-precision depth map;

[0034] A three-dimensional generation module is used to generate a three-dimensional point cloud for the high-precision depth map, and construct a three-dimensional model of the given image based on the three-dimensional point cloud.

[0035] A machine device, comprising:

[0036] processor; and

[0037] A memory having computer-readable instructions stored therein, wherein the computer-readable instructions are executed by the processor to implement the aforementioned method.

[0038] A computer-readable storage medium stores a computer program thereon, wherein the computer program implements the above-mentioned method when executed by a processor.

[0039] The technical solutions provided by the embodiments of the present invention may have the following beneficial effects:

[0040] In the three-dimensional reconstruction of a given image, first, in the feature extraction of the given image through each layer of the deep network, the feedforward network information of the integrated merging layer is mapped to the feedback network to perform feature extraction of the convolution layer, and the two-dimensional features of the given image are obtained. The mapping of the feedforward network information to the feedback network makes it unnecessary to store the information required by the feedback network, but directly calculate it, thereby greatly reducing memory consumption; secondly, after obtaining the two-dimensional features, sparse cost control is performed on the two-dimensional features, and then the two-dimensional features are transformed into three-dimensional features to obtain the cost information corresponding to the three-dimensional features; through the three-dimensional features Contextual information is extracted by mapping the cost volume information, an initial depth map predicted for a given image is generated, the initial depth map is propagated and optimized, a high-precision depth map is output, and finally a three-dimensional point cloud is generated for the high-precision depth map. A three-dimensional model of the given image is constructed from the three-dimensional point cloud. In the implementation of this three-dimensional reconstruction, the feature extraction of each layer of the deep network reduces memory consumption, and under the sparse cost control and mapping of the cost volume information, the amount of calculation is reduced to ensure the three-dimensional reconstruction effect. The complexity of the three-dimensional reconstruction is controlled, and the speed requirements of the three-dimensional reconstruction can be met. The length of time for processing a given image can meet the real-time requirements.

[0041] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0043] Figure 1 is a schematic diagram of an implementation environment according to the present invention;

[0044] Figure 2 is a hardware structure block diagram of a server according to an exemplary embodiment;

[0045] Figure 3 is a flow chart showing a three-dimensional reconstruction method according to an exemplary embodiment;

[0046] Figure 4 is based on Figure 3 A flowchart describing step 310 is shown in the corresponding embodiment;

[0047] Figure 5 is based on Figure 3 A flowchart describing step 313 is shown in the corresponding embodiment;

[0048] Figure 6 is a given picture input to the deep network according to an exemplary embodiment;

[0049] Figure 7 yes Figure 6 The feature map obtained by passing the given image through the 8-layer CNN network structure;

[0050] Figure 8 yes Figure 6 The feature map obtained by passing the given image through the 11-layer CNN network structure;

[0051] Figure 9 is based on Figure 3 A flowchart describing step 330 is shown in the corresponding embodiment;

[0052] Figure 10 This is a flowchart illustrating the steps of mapping input cost information through convolutional gates set in each layer of a network constructed for context information extraction to extract context information and generate an initial depth map for a given image prediction according to an exemplary embodiment;

[0053] Figure 11 1 is a schematic diagram of a recurrent neural network GRU structure according to an exemplary embodiment;

[0054] Figure 12 is based on Figure 3 A flowchart describing step 370 is shown in the corresponding embodiment;

[0055] Figure 13 The figure is a block diagram of a 3D reconstruction system according to an exemplary embodiment. DETAILED DESCRIPTION

[0056] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0057] Over the decades of development, 3D reconstruction technology has matured to the point where geometric modeling is the most mature, but the process is complex and time-consuming. While 3D reconstruction using 3D scanners can accurately generate 3D models, it requires expensive equipment, making it uneconomical. Furthermore, the equipment's operating conditions are demanding, making it unsuitable for use in simple environments. Therefore, neither geometric modeling nor 3D reconstruction using 3D scanners is suitable for 3D reconstruction applications, necessitating a new approach to achieving 3D reconstruction within computer vision technology.

[0058] With the continuous development of deep learning, computer vision technology has begun to merge with deep learning, providing image-based deep learning methods. Using deep learning to achieve 3D reconstruction is a promising approach, addressing the challenges of both geometric modeling and 3D reconstruction using 3D scanners. However, practical applications still face challenges, such as significant memory consumption, computational complexity, and long processing times, which limit the application of deep learning-based 3D reconstruction.

[0059] In response to many existing problems, the present invention will optimize the network structure and provide a three-dimensional reconstruction method with deep network optimization and low computational complexity, thereby reducing redundancy and enhancing timeliness.

[0060] Figure 1 FIG2 is a schematic diagram of an implementation environment according to the present invention. In an exemplary embodiment, the implementation environment includes an image source 110 and a server 130. For images captured by the image source 110, the server 130 performs 3D reconstruction of the images to obtain a 3D model.

[0061] For example, Figure 1 As shown, the image source 110 may be a user terminal held by a user, and the image is captured by a camera in the user terminal.

[0062] The implementation environment includes at least an image source 110 and a server 130. These two systems work together to perform 3D reconstruction of input images. The specific framework of this implementation environment is highly dependent on the specific scenario. Different scenarios will require different architectures, in addition to the image source 110 and server 130.

[0063] For example, this implementation environment can provide a service interface for 3D reconstruction required by various applications. For applications that require 3D reconstruction, it is only necessary to build a 3D reconstruction platform through the server and encapsulate the callable API interface.

[0064] Therefore, the server 130 will face the devices where various applications are located, and realize the three-dimensional reconstruction of the images in the applications through the interaction between the two and the interface call.

[0065] In addition, there are many other scenarios, such as single image recognition in video surveillance and human-computer interaction, which are not listed here one by one.

[0066] It should be understood that the description of this implementation environment is merely an illustration of an implementation scenario, but is not limited thereto. For any given picture, a three-dimensional model can be obtained by means of the three-dimensional reconstruction of the present invention.

[0067] Figure 2 This is a hardware structure block diagram of a server 200 according to an exemplary embodiment. It should be noted that the server 200 is only an example adapted for the present disclosure and cannot be considered to provide any limitation on the scope of use of the present disclosure. The server 200 cannot be interpreted as needing to rely on or must have Figure 2 One or more components of the exemplary server 200 shown in FIG.

[0068] The hardware structure of the server 200 may vary greatly due to different configurations or performances, such as Figure 2 As shown, the server 200 includes a power supply 210 , an interface 230 , at least one storage medium 250 , and at least one central processing unit (CPU) 270 .

[0069] The power supply 210 is used to provide operating voltage for each hardware device on the server 200 .

[0070] The interface 230 includes at least one wired or wireless network interface 231 , at least one serial-to-parallel conversion interface 233 , at least one input / output interface 235 , and at least one USB interface 237 , etc., for communicating with external devices.

[0071] The storage medium 250 is a carrier for resource storage, which can be a random access memory medium, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method can be temporary storage or permanent storage. Among them, the operating system 251 is used to manage and control the hardware devices and application 253 on the server 200 to enable the central processing unit 270 to calculate and process the massive data 255. It can be Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, etc. The application 253 is a computer program that performs at least one specific task based on the operating system 251, which can include at least one module ( Figure 2 (not shown in the figure), each module may respectively include a series of operating instructions for the server 200. The data 255 may be photos, pictures, etc. stored in a disk.

[0072] The central processing unit 270 may include one or more processors and is configured to communicate with the storage medium 250 via a bus for computing and processing the massive data 255 in the storage medium 250 .

[0073] As described above in detail, the server 200 applicable to the present disclosure performs three-dimensional reconstruction by having the central processing unit 270 read a series of operation instructions stored in the storage medium 250 .

[0074] Figure 3 FIG. 1 is a flow chart of a three-dimensional reconstruction method according to an exemplary embodiment. In an exemplary embodiment, the three-dimensional reconstruction method, such as Figure 3 As shown, it at least includes the following steps.

[0075] In step 310, in the feature extraction of the given image through each layer of the deep network, the feedforward network information performed by the integrated merging layer is mapped to the feedback network to perform feature extraction of the convolution layer to obtain the two-dimensional features of the given image.

[0076] In step 330 , sparse cost control is performed on the two-dimensional features, and the two-dimensional features are transformed into three-dimensional features to obtain cost information corresponding to the three-dimensional features.

[0077] In step 350 , context information is extracted by mapping the cost information corresponding to the three-dimensional features to generate an initial depth map predicted for the given image.

[0078] In step 370 , the initial depth map is propagated and optimized to output a high-precision depth map.

[0079] In step 390 , a three-dimensional point cloud is generated for the high-precision depth map, and a three-dimensional model of the given image is constructed from the three-dimensional point cloud.

[0080] At this point, the entire 3D reconstruction process is completed through the exemplary steps described above. Under this architecture, on the one hand, by integrating the merging layer into the deep network, the calculated part of the feedforward network information does not need to occupy memory, and through reversible calculation, it is mapped to the feedback network, that is, the unstored part of the feedforward network information is restored, and then the feature extraction of the convolution layer is performed, saving the required memory and achieving lightweight feature extraction;

[0081] On the other hand, sparse cost control will be implemented to optimize calculations, making feature extraction and depth map acquisition lightweight, ultimately achieving a three-dimensional representation of the image.

[0082] Specifically, during step 310, a given image corresponds to a 3D reconstructed object. Using the constructed deep network as the backbone, the deep network performs feature extraction on the given image, generating a rich 2D feature representation, i.e., the 2D features of the given image. It should be noted that the given image is a 2D description of the 3D reconstructed object. Therefore, the given image contains several 2D features, which represent the 3D reconstructed object in different dimensions.

[0083] The given image used for 3D reconstruction and thus obtaining a 3D model is an original image of the 3D reconstructed object. In one exemplary embodiment, the given image used is not limited to a single original image of the 3D reconstructed object; it may also be a sequence of original images of the 3D reconstructed object, which increases memory consumption. Therefore, during step 310, lightweight feature extraction is performed on the given image, significantly improving memory management efficiency.

[0084] The convolution layer of the deep network is a two-dimensional feature extraction layer, which takes a given image as input and extracts the features of the given image. By analogy, each layer of the deep network will continuously perform feature extraction through the constructed convolution layer to generate two-dimensional features that describe the three-dimensional reconstructed object in two dimensions under different dimensions. On this basis, the merging layer is integrated into each layer of the deep network, and the combination of the merging layer and the convolution layer constitutes the network layer that performs feature extraction in the deep network, so that the feature extraction of the convolution layer can be performed by mapping the feedforward network information performed by the merging layer to the feedback network during the execution of step 310.

[0085] At this point, it should be noted that without changing the convolution kernel, as the number of convolution layers increases, the memory consumption will increase. Therefore, the integration of the merging layer will reduce the memory consumption brought about by the increase in convolution layers, that is, curb the memory consumption brought about by several convolution layers.

[0086] In an exemplary embodiment, the number of convolutional layers in the deep network is 8, thereby constructing a convolutional network for feature extraction from the deep network. Exemplarily, in this convolutional network, the stride of the 3rd and 6th convolutional layers is set to 3, and the network output channels is 32.

[0087] During step 310, the network integrates a merging layer, which coordinates with the convolutional layer. This merging layer then maps the feedforward network information from the merging layer to the feedback network to extract features from the convolutional layer. The feedforward network information, as an intermediate result, is then used by the feedback network and is a significant contributor to memory consumption during feature extraction.

[0088] The mapping of feedforward network information to the feedback network is the process of obtaining the feedback network from the feedforward network information calculated by the feedforward network. It should be noted that in this application, the feedforward network information calculated by the feedforward network is used to construct the feedback network. Among them, for the feedforward network information calculated by the feedforward network, a part of it is generated and stored locally with the calculation of the feedforward network, which can also be referred to as the information stored by the feedforward network; the other part of the feedforward network information calculated no longer occupies memory, thereby saving memory consumption.

[0089] The network layers for feature extraction, the merging layer and the convolution layer cooperate with each other. First, the forward propagation process is performed through the feedforward network in the merging layer, and the part of the feedforward network information calculated through the forward propagation process of the feedforward network is stored in the memory; for the corresponding feedback network, the back propagation process that needs to be performed will read part of the stored feedforward network information from the memory, and map the read part of the feedforward network information to the feedback network, and then perform feature extraction of the convolution layer, and finally output the two-dimensional features corresponding to the convolution layer through the feedback network.

[0090] At this point, the lightweight feature extraction implemented in step 310 is completed. For the three-dimensional reconstruction of a given image, the memory consumption is reduced and rich two-dimensional features are obtained.

[0091] After obtaining the two-dimensional features for a given image, sparse cost control is performed through step 330. The sparse cost control refers to performing cost measurement and sparse operations on the two-dimensional features to generate a sparse cost, thereby performing the conversion of the two-dimensional features to three-dimensional features for the given image under the control of the sparse cost, and generating cost information corresponding to the three-dimensional features.

[0092] It should be understood that the cost information corresponding to a 3D feature is a 3D description of the 3D reconstructed object in a given image, measured in terms of cost. Therefore, sparse cost control aims to reduce computational complexity and memory consumption by converting 2D features into 3D features, while also controlling and ensuring the quality of 3D reconstruction.

[0093] As previously mentioned, the cost information corresponding to the 3D features is used to control the extraction of information from the 3D description of a given image, thereby enabling controllable and better preservation of feature information while significantly reducing the amount of computation required. The information extracted from the 3D description of a given image is used to obtain a depth map, which is the 3D description of the given image. In other words, the cost information corresponding to the 3D features controls the extraction of information, and the initial depth map is generated from the extracted information.

[0094] The extracted information is context information, which is extracted for depth map prediction under the control of cost information, so that the generation of the depth map can ensure the controllability of computational complexity and quality.

[0095] The obtained initial depth map is used to generate a high-precision depth map by executing step 370. Compared with the initial depth map, the high-precision depth map has higher quality, which prepares for the subsequent generation of high-precision three-dimensional point cloud data.

[0096] For the high-precision depth map generated by the deep network, step 390 is executed to generate a three-dimensional point cloud. The three-dimensional point cloud is well-performed in details, especially in areas where features change significantly, such as letters and folds of clothing on the three-dimensional reconstructed object, as well as details that are not missing. Since the generated three-dimensional point cloud is relatively dense, it is able to construct a three-dimensional model for a given image.

[0097] The 3D model constructed from the 3D point cloud is a 3D graphical representation of the 3D reconstructed object generated from a given 2D image. The 3D reconstruction performed through steps 310 to 390 above utilizes a lightweight 3D reconstruction neural network architecture. This architecture ensures accuracy by converting the 3D point cloud into contextual information and a high-precision depth map. While maintaining accuracy, it significantly reduces memory consumption and runtime, thereby ensuring and enhancing the real-time performance of the 3D reconstruction.

[0098] Figure 4 is based on Figure 3 The flowchart of step 310 is shown in the corresponding embodiment. In an exemplary embodiment, the combination of the merging layer and the convolution layer constitutes the network layer that performs feature extraction in the deep network, such as Figure 4 As shown, step 310 includes:

[0099] In step 311, in the feature extraction of each network layer in the deep network, for the back propagation from the feedforward network to the feedback network in the merging layer, the information stored in the feedforward network is reversed for forward propagation calculation to restore the information required for back propagation.

[0100] In step 313, the convolutional features extracted by the convolutional layer for the given image are input into a feedback network for recovering the information required for back propagation, and the feedback network outputs the two-dimensional features corresponding to the convolutional layer.

[0101] Each layer in a deep network that performs feature extraction is a combination of merging and convolutional layers. This means that the merging layers work in conjunction with the convolutional layers to perform feature extraction while simultaneously enhancing memory management efficiency and reducing memory consumption. Specifically, during the feature extraction process performed continuously by each network layer, the backpropagation from the feedforward network to the feedback network in the merging layer of that network layer reads the information stored in the feedforward network from memory and performs a reverse forward propagation calculation on it to recover the information required for backpropagation.

[0102] It should be noted that the calculation of the merging layer includes the forward propagation process of the feedforward network and the backward propagation process of the feedback network. The network parameters calculated during the forward propagation process, that is, the information stored in the feedforward network, are part of the information calculated during the forward propagation process. They are used to recover the network parameters required for the backward propagation process, that is, the information required for the backward propagation process.

[0103] The feedback network of the merging layer in the network layer is obtained by feedback propagation of the required information. Back propagation is performed through this feedback network, and the two-dimensional features extracted from the network layer are ultimately output. That is, as described in step 313, for feature extraction at each layer in the deep network, the convolutional features extracted by the convolutional layer are input into the feedback network that has restored the information required for back propagation, and then the two-dimensional features are obtained through the feedback network.

[0104] At this point, we can see that for the network layers performing feature extraction, the convolutional layers are integrated with merging layers. On the one hand, compared with the original convolutional layer implementation, the reused BN layer + activation layer is converted into a merging layer, which can also be called an INPLACE-ABN layer, thereby constructing a combination of merging layers and convolutional layers on the network structure. This structure saves memory consumption by reducing the storage of intermediate results, that is, the information stored in the feedforward network. The information stored in the feedforward network is mapped to the information required in the feedback network. Therefore, it only increases the amount of computation by 0.8% to 2%, but saves 50% of memory consumption.

[0105] Further, Figure 5 is based on Figure 3 The flowchart of step 313 is shown in the corresponding embodiment. In an exemplary embodiment, as Figure 5 As shown, step 311 includes:

[0106] In step 401, the feature extraction of the deep network for a given image is performed by forward propagation through the feedforward network in the merging layer and then input to the convolutional layer, which performs feature extraction;

[0107] In step 403, a forward propagation flip calculation is performed on the information obtained by the forward propagation process and stored by the feedforward network, and the information required for the reverse propagation is restored through the flip calculation.

[0108] As previously mentioned, each layer in a deep network extracts features from a given image. This extraction is performed layer by layer, providing rich two-dimensional features for obtaining the corresponding 3D model of the reconstructed object. In this implementation, feature extraction from a given image begins by forward propagating through the feedforward network in the merging layer. This information is then fed into the convolutional layer, which performs feature extraction according to the network structure.

[0109] After the convolutional layer performs feature extraction, the resulting convolutional features are then back-propagated through the feedback network to output two-dimensional features. To do this, a reversible calculation—the reverse calculation of the forward propagation—is performed on the information calculated by the feedforward network and stored in memory to recover the information needed for back-propagation, resulting in the feedback network that will perform back-propagation for the convolutional features.

[0110] The feedback network recovers some of the discarded intermediate results and required parameters, that is, the information required for back propagation, through reversible calculation during back propagation, that is, the flipping and inversion of forward propagation, thereby improving the memory management performance in deep networks and reducing memory requirements.

[0111] It should be noted that the integration of the merging layer in the deep network framework allows only the most effective parameters to be stored in the calculation, that is, the parameters that backpropagation depends on. For the subsequent backpropagation required, only the inverted recovery calculation is performed and the forward propagation is reversed to obtain the information required for backpropagation.

[0112] At this point, for feature extraction, the deep network uses a convolutional neural network to extract features from a given image. Compared with residual networks, the complexity is significantly reduced, and it can meet the requirements of speed and efficiency.

[0113] In an exemplary embodiment, the deep network for feature extraction is an 8-layer convolutional neural network, in which the stride of the 3rd and 6th layers of convolution is set to 2, and an image of size H×W is used as input, that is, given an image, the final result is The feature map of size, that is, two-dimensional feature.

[0114] The network output channel is 32, and the specific network structure is shown in the following table:

[0115]

[0116] Table 1

[0117] Among them, conv is the convolutional layer, and Inplace-ABN is the integrated merging layer.

[0118] This 8-layer CNN network structure can also be replaced with an 11-layer network structure. However, the increase in the number of layers does not produce more details in feature extraction. Figure 6 is a given picture input to the deep network according to an exemplary embodiment, Figure 7 yes Figure 6 The feature map obtained by passing the given picture through the 8-layer CNN network structure, Figure 8 yes Figure 6 The feature map obtained by passing the given image through the 11-layer CNN network structure.

[0119] Therefore, in this exemplary embodiment, the 8-layer CNN network structure is better, and compared with the 11-layer network structure, the 8-layer CNN convolutional network with 32 output channels has only half of the convolution kernel parameters. The following table compares the 8-layer CNN network structure and the 11-layer CNN network structure.

[0120]

[0121] Table 2

[0122] Figure 9 is based on Figure 3 The flowchart of step 330 is shown in the corresponding embodiment. In an exemplary embodiment, step 330 is as follows Figure 9 As shown, the following steps are included:

[0123] In step 331 , a cost amount is generated by cost amount measurement on the two-dimensional features and a sparse operation is performed to obtain a sparse cost amount. The sparse cost amount is used to perform capacity control on context information extraction of a given image.

[0124] In step 333 , the two-dimensional features are converted into three-dimensional features through the sparse cost amount, and the cost amount information corresponding to the three-dimensional features is obtained.

[0125] Given the image features corresponding to the image, i.e., the two-dimensional features calculated by the above steps, a cost quantity is generated for the features by executing a cost quantity measurement method. In one exemplary embodiment, the cost quantity measurement method used is:

[0126]

[0127] Where: F refers to the two-dimensional features of a given image, N is the feature dimension, and C is the cost.

[0128] The cost volume obtained by calculation is used for sparse operation to further reduce the amount of calculation and memory consumption, but it will not reduce the effect of 3D reconstruction.

[0129] It should be understood that in order to obtain the three-dimensional model corresponding to the three-dimensional reconstructed object, that is, to realize the three-dimensional reconstruction, in addition to the two-dimensional features, it is also necessary to obtain the depth features, which are obtained by extracting the context information of a given image. Therefore, the subsequent context information extraction is controlled by the sparse cost amount, thereby ensuring the effect of feature extraction.

[0130] After obtaining the sparse cost, the two-dimensional features are converted into three-dimensional features, that is, The cost of size is the cost information corresponding to the three-dimensional features of a given image, where H is the height of the image, W is the width of the image, C is the number of channels, and B is the batch size.

[0131] Furthermore, in an exemplary embodiment, Figure 3 In the corresponding embodiment, step 350 includes:

[0132] In each layer of the network built for context information extraction, the input cost information is mapped through the convolutional gates set in the network layer to extract the context information and generate an initial depth map predicted for a given image.

[0133] As previously mentioned, the sparse cost volume obtained by executing step 330 is used to control the capacity of context information extraction for a given image. The extracted context information corresponding to the given image is used to generate an initial depth map predicted for the given image.

[0134] The network layers constructed to extract contextual information form a 3D convolutional network or a recurrent neural network (GRU). In one exemplary embodiment, a convolutional layer is added to each network layer to act as a convolution gate to map the cost information, serially transforming the unit from 2D to 3D along the depth direction to obtain a 3D visual space.

[0135] Figure 10 This is a flowchart describing the steps of mapping input cost information through convolutional gates set in each layer of a network constructed for context information extraction to extract context information and generate an initial depth map for predicting a given image according to an exemplary embodiment.

[0136] In an exemplary embodiment, the convolution gate is the first convolution gate. In each layer of the network constructed for context information extraction, the input cost information is mapped through the convolution gate set in the network layer to extract the context information to generate the initial depth map step for the given picture prediction, such as Figure 10 Shown, including:

[0137] In step 501, the configured convolution layer is used as the first convolution gate to map the cost information corresponding to the three-dimensional feature to the visual space;

[0138] In step 503, the mapping of the cost information in the visual space is combined with the reset gate, the update gate, and the second convolution gate configured by the update gate to extract context information of the influence control of the previous step, and generate an initial depth map predicted for the given image.

[0139] The GRU recurrent neural network, which generates an initial depth map for a given image, performs depth-wise regularization to generate a two-dimensional cost map for the cost information. Based on this, a 2D-to-3D unit transformation is performed to map the cost information into visual space. To this end, a convolutional layer is added to the GRU recurrent neural network, serving as the first convolutional gate to map the cost information.

[0140] At the same time, a convolution gate is added between the update gate and the activation function as the second convolution gate to enhance the regularization ability.

[0141] Specifically, the cost information of each channel will be used as input, and the update gate is used to update the reset information. After the cost information corresponding to the three-dimensional feature is mapped through the first convolution gate, the context information will be extracted from the top under the control of the second convolution gate.

[0142] Figure 11 : is a schematic diagram of a recurrent neural network GRU structure according to an exemplary embodiment. Figure 11 As shown, in conjunction with the context information extraction controlled by the second convolutional gate, the reset gate indicates the input information, that is, how the information of the previous step maps the input of this step. Under the control of the reset gate, the activation function σ and the update gate, the second convolutional gate performs context information extraction on the mapping of the cost volume information in the visual space and finally generates an initial depth map. At this point, the feature information can be better preserved and the amount of computation is greatly reduced.

[0143] As Figure 11 As shown, the recurrent neural network GRU generates the initial depth map through a network structure composed of several recurrent layers. Specifically, first, the state h of the previous recurrent layer is t-1 and the input x of the current loop layer t To implement the update gate z of the recurrent neural network GRU result t =σ(W (z) x t +U (z) h t-1 ) and reset gate r t=σ(W (r) x t +U (r) h t-1 ), where W (z) and W (r) They are the weight matrices corresponding to the inputs of the update gate and the reset gate, U (z) and U (r) are the weight matrices of the hidden states of the update gate and reset gate respectively.

[0144] For a loop layer, after completing the update gate and reset gate update based on the previous loop layer and the current loop layer, the obtained gate signal z t and r t , that is, the input x of the current cycle layer t Perform splicing and mapping. The above-mentioned gating operation affects the final output. Its specific implementation process includes: First, for the state h of the previous loop layer t-1 , use reset gating to get the data h′ after “reset” t-1 =h t-1 ⊙r, then h′ t-1 and the input x of the current loop layer t Merge and splice, and finally use a tanh activation function to scale the data to the range of -1 to 1, that is: h′ t =tanh(Wx t +r t ⊙Uh′ t-1 ).

[0145] Specifically, in the entire 3D reconstruction process, 49×7 or 64×7 (indicating 49 or 64 camera positions and 7 different illuminations) images are used as input. The images are input in batches, with 3 images in each batch. The size of the input image is 640×512, and the number of depth planes is set to 48 to map it to a depth of 425mm to 921mm. Through the 8-layer feature extraction network, a 3×32×128×160 feature vector is extracted and converted into a 3×32×48×64×80 regularization matrix. The cost regularization operation is performed through the recurrent neural network GRU structure. A 3×1×128×160 coarse depth map is predicted through regularization. The coarse depth map is used as the input of the LM refinement layer, and a 3×1×640×480 depth map is obtained through the refinement layer. Finally, the generated depth map is input into the open source code fusibile to fuse the depth maps generated by the model at different perspectives for point cloud reconstruction.

[0146] Figure 12 is based on Figure 3 The flowchart of step 370 is shown in the corresponding embodiment. In an exemplary embodiment, as Figure 12 As shown, step 370 includes:

[0147] In step 371, a coarse optimization operation is performed on the initial depth map by propagating the initial depth map through the convolutional network.

[0148] In step 373 , the output of the coarse optimization operation is used as input to refine the initial depth map to generate a high-precision depth map.

[0149] The execution of the coarse optimization operation in step 371 provides input objects for the subsequent refinement operation. In one exemplary embodiment, the coarse optimization operation is implemented by a 15-layer convolutional network, where the first 8 layers are the same as the layers performing feature extraction, and the last 7 layers are convolutional networks with a convolution kernel of 3.

[0150] After performing a coarse optimization operation on the initial depth map, a coarse depth map is obtained, and a refinement operation is performed on the coarse depth map to generate a high-precision depth map. It should be noted that the execution of this refinement operation is based on the update rule of the depth map refinement implemented by the LM algorithm, namely:

[0151]

[0152] in, is the value of point p on the dense depth map;

[0153] To this end, the incremental expression required is defined, namely:

[0154]

[0155] Among them, μ: affects the increment size and direction, and its expression is

[0156] J: refers to the Jacobian matrix of the difference between the original image and the reference image for the refined depth e is the unit matrix; i is the serial number of the current image; ξ is

[0157] At this point, after generating a high-precision depth map, a three-dimensional point cloud can be generated for it. The generated three-dimensional point cloud is intact in details and the point cloud is relatively dense, meeting the requirements of reconstruction.

[0158] Through the exemplary embodiments described above, an end-to-end, deep learning-based, lightweight 3D reconstruction network is implemented. This improves 3D reconstruction efficiency, from image to 3D point cloud extraction, while ensuring accuracy while significantly reducing memory consumption and runtime, ensuring real-time 3D reconstruction. The 3D reconstruction implemented by the present invention is a lightweight neural network that can be applied to public datasets. In today's environment of expensive computing resources, implementing a lightweight 3D reconstruction neural network architecture can effectively improve 3D reconstruction efficiency, effectively addressing the problem of deep learning networks being unable to be applied in large-scale scenarios, and providing excellent guarantees for neural network transplantation.

[0159] The following is an embodiment of the device of the present invention, which is used to perform the above-mentioned 3D reconstruction method embodiment of the present invention. For details not disclosed in the embodiment of the device of the present invention, please refer to the embodiment of the 3D reconstruction method of the present invention.

[0160] Figure 13 FIG. 1 is a block diagram of a 3D reconstruction system according to an exemplary embodiment. In an exemplary embodiment, Figure 13 As shown, the 3D reconstruction system includes but is not limited to: a feature construction module 710 , a sparse cost control module 730 , a depth prediction module 750 , an accuracy optimization module 770 and a 3D generation module 790 .

[0161] A feature construction module 710 is configured to extract features of a given image by mapping the feedforward network information of the integrated merging layer to the feedback network to perform convolutional layer feature extraction in the feature extraction of the given image through each layer of the deep network, thereby obtaining a two-dimensional feature of the given image;

[0162] a sparse cost control module 730 for performing sparse cost control on the two-dimensional features, and transforming the two-dimensional features into three-dimensional features to obtain cost information corresponding to the three-dimensional features;

[0163] A depth prediction module 750 is configured to extract context information by mapping the cost information corresponding to the three-dimensional features to generate an initial depth map predicted for the given image;

[0164] A precision optimization module 770 is configured to propagate and optimize the initial depth map and output a high-precision depth map;

[0165] The three-dimensional generation module 790 is used to generate a three-dimensional point cloud for the high-precision depth map, and construct a three-dimensional model of the given image based on the three-dimensional point cloud.

[0166] Optionally, the present invention also provides a machine device, which can be used for Figure 1 In the implementation environment shown, execute Figure 3 、 Figure 4 、 Figure 5 、 Figure 10 and Figure 12 All or part of the steps of any of the methods shown. The device includes:

[0167] processor;

[0168] a memory for storing processor-executable instructions;

[0169] The processor is configured to execute the aforementioned method.

[0170] The specific manner in which the processor of the device in this embodiment performs operations has been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0171] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: In the feature extraction of a given image through each layer of the deep network, the feedforward network information performed by the integrated merging layer is mapped to the feedback network to perform feature extraction of the convolution layer, thereby obtaining a two-dimensional feature of the given image. The combination of the merging layer and the convolution layer constitutes the network layer that performs feature extraction in the deep network; The method includes: in the feature extraction of a given image through each layer of the deep network, mapping the feedforward network information performed by the integrated merging layer to the feedback network to perform feature extraction of the convolution layer, thereby obtaining a two-dimensional feature of the given image, including: in the feature extraction of each layer of the deep network, in the back propagation from the feedforward network in the merging layer to the feedback network, flipping the information stored in the feedforward network to perform forward propagation calculation to restore the information required for the back propagation; inputting the convolution features extracted by the convolution layer for the given image into the feedback network that restores the information required for the back propagation, and outputting the two-dimensional features corresponding to the convolution layer in the feedback network; Performing sparse cost control on the two-dimensional features and transforming the two-dimensional features into three-dimensional features to obtain cost information corresponding to the three-dimensional features; Mapping the cost information corresponding to the three-dimensional feature to the visual space using the configured convolutional layer as the first convolution gate; Mapping the cost information in the visual space, extracting context information from the previous step's impact control in conjunction with a reset gate, an update gate, and a second convolution gate configured by the update gate, to generate an initial depth map predicted for the given image; Propagating and optimizing the initial depth map to output a high-precision depth map; A three-dimensional point cloud is generated for the high-precision depth map, and a three-dimensional model of the given image is constructed based on the three-dimensional point cloud.

2. The method according to claim 1, characterized in that In the feature extraction of each layer of the deep network, for the back propagation from the feedforward network to the feedback network in the merging layer, the information stored in the feedforward network is reversed for forward propagation calculation to restore the information required for the back propagation, including: Extracting features of the given image in the deep network, forward propagating through the feedforward network in the merging layer and then inputting them into the convolutional layer, where the convolutional layer performs feature extraction; The information obtained by the forward propagation process and stored by the feedforward network is subjected to the forward propagation flip calculation, and the information required for the reverse propagation is recovered through the flip calculation.

3. The method according to claim 1, characterized in that The performing sparse cost control on the two-dimensional features and transforming the two-dimensional features into three-dimensional features to obtain cost information corresponding to the three-dimensional features includes: Generating a cost amount by cost amount measurement on the two-dimensional feature and performing a sparse operation to obtain a sparse cost amount, wherein the sparse cost amount is used to perform capacity control on context information extraction of the given image; The two-dimensional features are converted into three-dimensional features through the sparse cost amount to obtain cost amount information corresponding to the three-dimensional features.

4. The method according to claim 1, wherein The propagation and optimization of the initial depth map to output a high-precision depth map includes: Performing a coarse optimization operation on the initial depth map by propagating the initial depth map through a convolutional network; The output of the coarse optimization operation is used as an input object, and the initial depth map is refined to generate a high-precision depth map.

5. A three-dimensional reconstruction system, characterized in that: The system comprises: A feature construction module is used to extract features of a given image in a deep network through each network layer, by mapping the feedforward network information performed by the integrated merging layer to the feedback network to perform convolutional layer feature extraction, thereby obtaining two-dimensional features of the given image. The combination of the merging layer and the convolution layer constitutes the network layer that performs feature extraction in the deep network; The feature construction module is further used to, in the feature extraction of each layer of the network in the deep network, reverse the forward propagation calculation of the information stored in the feedforward network to the feedback network in the merging layer to restore the information required for the back propagation; input the convolutional features extracted by the convolutional layer for a given image into the feedback network that restores the information required for the back propagation, and output the two-dimensional features corresponding to the convolutional layer in the feedback network; a sparse cost control module, configured to perform sparse cost control on the two-dimensional features, and transform the two-dimensional features into three-dimensional features to obtain cost information corresponding to the three-dimensional features; A depth prediction module, configured to map the cost information corresponding to the three-dimensional feature to the visual space using the configured convolution layer as a first convolution gate; Mapping the cost information in the visual space, extracting context information from the previous step's impact control in conjunction with a reset gate, an update gate, and a second convolution gate configured by the update gate, to generate an initial depth map predicted for the given image; A precision optimization module, configured to propagate and optimize the initial depth map and output a high-precision depth map; A three-dimensional generation module is used to generate a three-dimensional point cloud for the high-precision depth map, and construct a three-dimensional model of the given image based on the three-dimensional point cloud.

6. A machine device, characterized in that: include: processor; and A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions are executed by the processor to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images