A three-dimensional reconstruction method, device and medium based on adaptive network

Abstract

The present invention discloses a three-dimensional reconstruction method, device and medium based on an adaptive network, wherein the method comprises: inputting two images, using two feature extraction networks to extract features from the two images respectively, using an adaptive network to process the extracted features, and obtaining two image feature maps; constructing a cost body of the view according to the image feature map and camera parameters, and obtaining two cost bodies; using a regularization network to perform regularization processing on all cost bodies, so as to utilize spatial context information and convert the cost into a probability distribution of a depth hypothesis; using a loss function to train the feature extraction network, the adaptive network and the regularization network, and obtaining a three-dimensional reconstruction model, and using the obtained three-dimensional reconstruction model to perform three-dimensional reconstruction on the processed image. The present invention uses an adaptive network to process features, and uses a regularization network to regularize the cost body, so as to effectively improve the accuracy of three-dimensional reconstruction, and can be widely used in the field of three-dimensional reconstruction technology.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method, device and medium based on an adaptive network. Background Art

[0002] The 3D reconstruction technology based on deep learning mainly uses the convolutional neural network (CNN) as a learning model. It learns the 3D information of objects in the image through a large amount of data and realizes the end-to-end conversion of 2D image objects to 3D object models. Compared with traditional 3D reconstruction methods, deep learning methods have great advantages and have achieved significant improvements compared with other algorithms. Applying deep learning technology to 3D reconstruction has natural advantages. Deep learning can use prior experience to roughly estimate the depth of objects. Its principle is similar to that of humans roughly estimating the distance of objects based on experience, which has certain rationality. Therefore, introducing deep learning into 3D reconstruction, relying on its ability to automatically learn features, and using neural networks instead of stereo matching has achieved good results and has become the current mainstream idea. In recent years, a large number of excellent 3D reconstruction algorithms based on deep learning have emerged.

[0003] Although people have begun to study 3D reconstruction methods based on deep learning in recent years, there are still some basic problems. First of all, in terms of high precision, although the current technology can restore rough 3D object models and continuously improve the resolution of reconstruction, its accuracy still has not reached a sufficiently precise level. On the surface of objects with richer object information, the accuracy of 3D reconstruction still faces major challenges.

[0004] Secondly, in the face of complex outdoor scenes, how to obtain more accurate object information without being disturbed by outdoor replication factors, so as to reconstruct a 3D model, also needs to be studied and solved, especially in the face of occluded, textureless or low-textured objects, how to face this challenge, so as to obtain a more refined 3D reconstruction model, has always been a difficulty that needs to be faced. In addition, at present, algorithms based on deep learning rely more on traditional algorithms, and improve some of the traditional algorithms, and their versatility is poor.

[0005] Based on the above difficult problems, further research is needed on how to use deep learning methods to improve the accuracy of three-dimensional reconstruction and reduce the complexity of the algorithm. Summary of the invention

[0006] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a three-dimensional reconstruction method, device and medium based on an adaptive network.

[0007] The technical solution adopted by the present invention is:

[0008] A three-dimensional reconstruction method based on an adaptive network comprises the following steps:

[0009] Input two images, use two feature extraction networks to extract features from the two images respectively, use an adaptive network to process the extracted features, and obtain two image feature maps;

[0010] Construct the cost volume of the view according to the image feature map and camera parameters, and obtain two cost volumes;

[0011] A regularization network is used to regularize all cost volumes to utilize spatial context information and transform the cost into a probability distribution of depth hypotheses.

[0012] The loss function is used to train the feature extraction network, the adaptive network and the regularization network to obtain a three-dimensional reconstruction model, and the obtained three-dimensional reconstruction model is used to perform three-dimensional reconstruction on the image to be processed.

[0013] Furthermore, the parameters of the two feature extraction networks are the same and the parameters are shared; the feature extraction network consists of a five-layer CNN.

[0014] Further, the adaptive network is used to adaptively aggregate features of regions of different scales and different texture richness;

[0015] There are three different feature space scales in the adaptive network, which are obtained by three deformable convolution processes:

[0016]

[0017] Among them, f(p) represents the eigenvalue of pixel p, w k represents the kernel parameters defined in the ordinary convolution operation, p k represents a fixed offset, Δp k and Δm k refers to the offset and modulation weights adaptively generated by the learnable sub-network of the deformable convolution.

[0018] Furthermore, constructing a cost volume of the view according to the image feature map and the camera parameters to obtain two cost volumes includes:

[0019] The features of the two images and the camera parameters are matched and calculated to obtain two cost volumes;

[0020] When the depth of the camera is assumed to be d, the pixel mapping relationship between the image features and the i-th input image is:

[0021]

[0022] Among them, T and K represent the external transformation parameters and internal parameters of the camera;

[0023] The expression of the cost body is as follows:

[0024]

[0025] Among them, f i (d) represents the features of the input i-th image, Represents the average features of two images.

[0026] Furthermore, after obtaining the cost body, the following steps are also included:

[0027] A convolutional network module is used to process unreliable matching cost bodies. The specific expression is as follows:

[0028]

[0029] Among them, * is the product of the corresponding elements of the matrix, and ω(·) is the cost volume of each image that adaptively generates features.

[0030] Furthermore, the regularized network adopts a hybrid RNN-CNN approach, and the substitution amount H×W×D×32 is cut at the D dimension;

[0031] The features passed through the regularization network have horizontal and vertical directions. In the horizontal direction, each piece of the three-dimensional cost volume is regularized by a CNN with an encoder-decoder architecture; in the vertical direction, 5 parallel RNNs are used to pass the intermediate output of the original convolutional layer to the next convolutional layer.

[0032] Furthermore, the five parallel RNNs are used to pass the intermediate output of the original convolutional layer to the next convolutional layer, including:

[0033] The cost volume of depth d that needs to be processed by the j-th convolutional layer (the output of the j-1th layer) is recorded as v j-1 (d), the output of this layer at depth d-1 is v j (d-1), at this time, keep the original value operation (that is, the hidden state) and record it as m j (d-1); v j-1 (d) and v j (d-1) is connected in series. After being processed by the convolution layer, the tensor of the feature dimension is divided into 4 tensors w, x, y, z. The 4 two-dimensional signals in a convolution layer are:

[0034]

[0035] Due to m j (d) = m j (d-1), we can get:

[0036] m j (d) = m j (d-1)·f+i·g

[0037] The final output cost volume v j The expression of (d) is as follows:

[0038] v j (d) = o·tanh(m j (d))

[0039] Among them, w, x, y, and z represent the four dimensional information of the four-dimensional matrix obtained by feature extraction.

[0040] Furthermore, since regularization converts the matching cost into a depth hypothesis of pixel-level probability distribution, the task of depth estimation is similar to the pixel-level classification problem. Therefore, the training loss is calculated by encoding the true value and using cross entropy. The specific expression is as follows:

[0041]

[0042] Among them, G d (p) and P d (p) represents the probability of the real image of pixel p and the predicted probability of the depth hypothesis d, p v is a valid pixel set of reliable depth; d0 represents the depth hypothesis prediction probability of the first image, and d1 represents the depth hypothesis prediction probability of the second image.

[0043] Another technical solution adopted by the present invention is:

[0044] A three-dimensional reconstruction device based on an adaptive network, comprising:

[0045] at least one processor;

[0046] at least one memory for storing at least one program;

[0047] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0048] Another technical solution adopted by the present invention is:

[0049] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to execute the method described above when executed by the processor.

[0050] The beneficial effects of the present invention are as follows: the present invention uses an adaptive network to process features and uses a regularization network to regularize the cost volume, thereby effectively improving the accuracy of three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 is a flow chart of a three-dimensional reconstruction method based on an adaptive network in an embodiment of the present invention;

[0053] Figure 2 is a schematic diagram of the structure of an adaptive network in an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the network structure of the aggregated cost body in an embodiment of the present invention;

[0055] Figure 4 is a schematic diagram of the cost body regularization network structure in an embodiment of the present invention;

[0056] Figure 5 It is a flowchart of the steps of a three-dimensional reconstruction method based on an adaptive network in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0058] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0059] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0060] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0061] Terminology explanation:

[0062] CNN: Convolutional Neural Network.

[0063] RNN: Recurrent Neural Network.

[0064] The current 3D reconstruction has limited robustness and integrity when dealing with occluded, textureless or low-textured local backgrounds, and most binocular vision matching cost aggregation processes generally do not consider pixel-level visibility, which reduces the accuracy of the 3D reconstruction model. Figure 1 and Figure 5 As shown, this embodiment provides a 3D reconstruction method based on an adaptive network, which specifically includes the following steps:

[0065] S1. Input two images, use two feature extraction networks to extract features from the two images respectively, use an adaptive network to process the extracted features, and obtain two image feature maps.

[0066] Feature extraction: The features of all images are obtained by passing the two input images through the feature extraction network. The parameters of the two feature extraction networks are the same, but each image needs to be feature extracted, and finally two image feature maps are obtained. The feature extraction network consists of five layers of CNN. The two input images are passed through this feature extraction network respectively to obtain the feature map. The parameters between the feature extraction networks are shared, and the convolution kernel size is 3*3.

[0067] This adaptive network can be used to adaptively aggregate features of regions of different scales and texture richness. In this network, there are three different feature space scales, which are obtained by three deformable convolutions:

[0068]

[0069] Where f(p) represents the eigenvalue of pixel p, w krepresents the kernel parameters defined in the ordinary convolution operation, p k represents a fixed offset, Δp k and Δm k It refers to the offset and modulation weights adaptively generated by the learnable sub-network of deformable convolution. By interpolating the smaller feature map to H×W, we can get 3 feature maps with 16, 8, and 8 channels respectively, and concatenate these features into a feature map of H×W×32.

[0070] The feature map obtained by the feature extraction network is input into the convolutional neural network. The convolution kernels of all convolutional layers in the network are 3×3. First, the feature channel sizes at the encoding are 8, 16, 16, 16, and 16. Then, the multi-scale feature map is passed through three deformable convolutions respectively, and then the values ​​obtained by the deformable convolution are processed by bilinear interpolation of different sizes. Three matrices with different channel sizes can be obtained. Finally, the matrices of different channels obtained in the previous step are concatenated to obtain a feature map of H×W×32 (16+8+8). See Figure 2 .

[0071] S2. Construct a cost volume of the view according to the image feature map and the camera parameters to obtain two cost volumes.

[0072] Aggregate cost volume: The matching cost volume for each view is calculated by matching the features of the two images with the camera parameters. When the depth is assumed to be d, the pixel mapping relationship between the image features and the i-th input image is:

[0073]

[0074] Where T and K represent the external transformation parameters and internal parameters of the camera. Based on this, the cost volume can be obtained:

[0075]

[0076] where f i (d) represents the features of the input i-th image, Represents the average features of two images.

[0077] After constructing the cost volume of each view, the next step is to aggregate all the cost volumes into one cost volume through regularization. A common approach is to average the cost volume, with the basic idea that all views should be equally important. However, this is not reasonable enough, because different shooting angles may cause problems such as occlusion or lighting effects, making depth estimation more difficult. Therefore, this embodiment uses a convolutional network module to handle unreliable matching cost volumes:

[0078]

[0079] The * is the product of the corresponding elements of the matrix, and the cost volume of each image ω(·) adaptively generates features. In this way, pixels that are easily confused or difficult to identify during the matching process are suppressed, and pixels with key contextual information will be given greater weight. 1+ω(·) is used here because it can better prevent over-smoothing than using ω(·) alone. See Figure 3 .

[0080] S3. Regularization network is used to regularize all cost volumes to utilize spatial context information and transform the cost into a probability distribution of depth hypothesis.

[0081] Cost volume regularization: Cost regularization uses spatial context information to convert the matching cost into a probability distribution of D depth hypotheses. The regularization network uses a hybrid RNN-CNN approach to slice the cost volume H×W×D×32 at the D dimension. The features passed through the regularization network have horizontal and vertical directions. In the horizontal direction, each slice of the three-dimensional cost volume is regularized by a CNN with an encoder-decoder architecture. In the vertical direction, there are 5 parallel RNNs that pass the intermediate output of the original convolution module to the next convolution module. See Figure 4 .

[0082] The cost volume of depth d that needs to be processed by the j-th convolutional layer (the output of the j-1th layer) is recorded as v j-1 (d), the output of this layer at depth d-1 is v j (d-1), at this time, keep the original value operation (that is, the hidden state) and record it as m j (d-1). j-1 (d) and v j (d-1) are connected in series. After being processed by the convolution layer, the feature dimension tensor is divided into 4 tensors w, x, y, z. The 4 two-dimensional signals in a convolution layer are:

[0083]

[0084] Then m j (d) = m j (d-1) can be obtained:

[0085] m j (d) = m j (d-1)·f+i·g

[0086] The final output cost of the network is v j (d) is:

[0087] v j (d) = o·tanh(mj (d))

[0088] S4. Use the loss function to train the feature extraction network, the adaptive network and the regularization network to obtain a three-dimensional reconstruction model, and use the obtained three-dimensional reconstruction model to perform three-dimensional reconstruction on the image to be processed.

[0089] Loss function: Since the cost volume regularization transforms the matching cost into a depth hypothesis with pixel-level probability distribution, the task of depth estimation is now similar to a pixel-level classification problem. Therefore, by encoding the true value, cross entropy is used to calculate the training loss, which is defined as:

[0090]

[0091] In the formula, G d (p) and P d (p) represents the probability of the real image of pixel p and the predicted probability of the depth hypothesis d, p v is the set of valid pixels for reliable depth.

[0092] To better describe the method of this embodiment, a specific example is used below to illustrate.

[0093] The DTU dataset is an indoor MVS dataset with a fixed camera trajectory under well-controlled laboratory conditions. It contains 128 scans, 49 views under 7 different lighting conditions, and is divided into 79 training scans, 18 validation scans, and 22 evaluation scans. With each image as a reference, there are a total of 27,097 training samples. According to the common configuration, the DTU dataset is used for network training and evaluation. Next, the specific process of the experiment and the results of using the above dataset are introduced, and compared with the current mainstream algorithms for verification. The smaller the comparison data used, the better the effect.

[0094] The final experimental results are shown in Table 1:

[0095] Table 1

[0096] method Acc.(mm) Comp.(mm) Overall(mm) 3D reconstruction average 0.386 0.369 0.377 This method 0.396 0.342 0.369

[0097] The results of this experimental method on the data set show that the accuracy is improved compared with other methods. This network is a good method for calculating binocular vision 3D point cloud reconstruction.

[0098] In summary, the method of this embodiment has the following advantages and beneficial effects compared with the prior art:

[0099] (1) An adaptive network scheme is designed for robust feature extraction, which adaptively aggregates context-aware features into texture regions of multiple scales and different richness.

[0100] (2) A convolutional network is used in the multi-view cost volume aggregation step, which aims to overcome the difficulty of different occlusions in complex scenes by assigning higher weights to well-matched view pairs.

[0101] This embodiment also provides a three-dimensional reconstruction device based on an adaptive network, comprising:

[0102] at least one processor;

[0103] at least one memory for storing at least one program;

[0104] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 5 The method shown.

[0105] A three-dimensional reconstruction device based on an adaptive network in this embodiment can execute a three-dimensional reconstruction method based on an adaptive network provided by an embodiment of the method of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0106] The present application also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes Figure 5 The method shown.

[0107] This embodiment also provides a storage medium, which stores instructions or programs that can execute a three-dimensional reconstruction method based on an adaptive network provided by an embodiment of the method of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0108] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0109] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0110] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0111] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0112] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0113] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0114] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0115] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0116] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A three-dimensional reconstruction method based on an adaptive network, characterized in that: The following steps are involved: Input two images, use two feature extraction networks to extract features from the two images respectively, use an adaptive network to process the extracted features, and obtain two image feature maps; Construct the cost volume of the view according to the image feature map and camera parameters, and obtain two cost volumes; A regularization network is used to regularize all cost volumes to utilize spatial context information and transform the cost into a probability distribution of depth hypotheses. The loss function is used to train the feature extraction network, the adaptive network and the regularization network to obtain a three-dimensional reconstruction model, and the three-dimensional reconstruction of the image to be processed is performed using the obtained three-dimensional reconstruction model; The cost volume of the view is constructed according to the image feature map and the camera parameters to obtain two cost volumes, including: The features of the two images and the camera parameters are matched and calculated to obtain two cost volumes; When the depth of the camera is assumed to be d, the pixel mapping relationship between the image features and the i-th input image is: H i (d)=d·K i ·T i ·T1 -1 ·K1 -1 Among them, T and K represent the external transformation parameters and internal parameters of the camera; The expression of the cost body is as follows: Among them, f i (d) represents the features of the input i-th image, represents the average features of two images; After obtaining the cost body, the following steps are also included: A convolutional network module is used to process unreliable matching cost bodies. The specific expression is as follows: Among them, * is the product of the corresponding elements of the matrix, and ω(·) is the cost volume of each image that adaptively generates features.

2. The three-dimensional reconstruction method based on an adaptive network according to claim 1, characterized in that: The parameters of the two feature extraction networks are the same and the parameters are shared; the feature extraction network consists of a five-layer CNN.

3. The three-dimensional reconstruction method based on an adaptive network according to claim 1, characterized in that: The adaptive network is used to adaptively aggregate features of regions with different scales and different texture richness; There are three different feature space scales in the adaptive network, which are obtained by three deformable convolution processes: Among them, f′(p) represents the eigenvalue of pixel p, w k represents the kernel parameters defined in the ordinary convolution operation, p k represents a fixed offset, Δp k and Δm k refers to the offset and modulation weights adaptively generated by the learnable sub-network of the deformable convolution.

4. The three-dimensional reconstruction method based on an adaptive network according to claim 1, characterized in that: The regularized network adopts RNN-CNN hybrid mode, and cuts the value H×W×D×32 at the D dimension; The features passed through the regularization network have horizontal and vertical directions. In the horizontal direction, each piece of the three-dimensional cost volume is regularized by a CNN with an encoder-decoder architecture; in the vertical direction, 5 parallel RNNs are used to pass the intermediate output of the original convolutional layer to the next convolutional layer.

5. The three-dimensional reconstruction method based on an adaptive network according to claim 4, characterized in that: The method uses 5 parallel RNNs to pass the intermediate output of the original convolutional layer to the next convolutional layer, including: The cost volume of depth d that the j-th convolutional layer needs to process is recorded as v j-1 (d), the output of this layer at depth d-1 is v j (d-1), at this time, the operation of keeping the original value is recorded as m j (d-1); v j-1 (d) and v j (d-1) is connected in series. After being processed by the convolution layer, the tensor of the feature dimension is divided into 4 tensors w, x, y, z. The 4 two-dimensional signals in a convolution layer are: Due to m j (d) = m j (d-1), we can get: m j (d)=m j (d-1)·f+i·g The final output cost volume v j The expression of (d) is as follows: v j (d)=o·tanh(m j (d))。 6. The method for three-dimensional reconstruction based on an adaptive network according to claim 1, characterized in that: Since regularization converts the matching cost into a depth hypothesis of pixel-level probability distribution, the task of depth estimation is similar to the pixel-level classification problem. Therefore, the training loss is calculated by encoding the true value and using cross entropy. The specific expression is as follows: Among them, G d (p) and P d (p) represents the probability of the real image of pixel p and the predicted probability of the depth hypothesis d, p v is a valid pixel set of reliable depth; d1 represents the depth hypothesis prediction probability of the second image, and d0 represents the depth hypothesis prediction probability of the first image.

7. A three-dimensional reconstruction device based on an adaptive network, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 6 when executed by the processor.

Images