Monocular depth estimation method and device integrating planar structure information

Abstract

The present invention provides a monocular depth estimation method and device integrating plane structure information, which relates to the field of computer vision technology, wherein the method comprises: constructing a depth estimation model, and training the depth estimation model according to a multi-task loss function based on contrastive learning and a sample scene image set with relative depth value labels and plane instance labels; the depth estimation model is composed of a feature extraction module, a plane structure instance segmentation module and a monocular depth estimation module; the target image is input into the trained depth estimation model to obtain a relative depth map corresponding to the target image. The method can assist in supervising the monocular depth estimation task through the plane structure information in the scene image, thereby improving the robustness and accuracy of the monocular depth estimation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a monocular depth estimation method and device integrating planar structure information. Background Art

[0002] Currently, monocular depth estimation is a popular technology in the field of computer vision. This technology has wide applications in scene understanding, three-dimensional reconstruction and other fields.

[0003] With the development of intelligent technology, existing monocular depth estimation methods are generally divided into two categories: traditional methods and deep learning methods. Among them, traditional methods are generally based on machine learning, using Markov Random Field (MRF) or Conditional Random Field (CRF) to model the mapping relationship between image features and depth, and solving the depth by minimizing the energy function under the maximum a posteriori probability framework. However, these features extracted from a single two-dimensional image cannot well represent the three-dimensional structural information of the scene, so the application effect is not ideal. Deep learning methods have brought profound changes to monocular depth estimation at multiple levels such as data processing, geometric inference, structural reasoning, and semantic understanding. This type of method generally requires the construction of a deep learning model to achieve depth estimation of a single scene image through supervised or unsupervised training. The deep learning method of monocular depth estimation has achieved good application results in ordinary scenes.

[0004] However, due to the systematic problem of insufficient single image input information, the current deep learning methods for monocular depth estimation lack robustness and accuracy, especially when used in indoor scenes with occlusion and perspective changes, their robustness and accuracy are significantly reduced. Summary of the invention

[0005] In view of the problems existing in the prior art, an embodiment of the present invention provides a monocular depth estimation method and device integrating planar structure information.

[0006] The present invention provides a monocular depth estimation method integrating planar structure information, comprising:

[0007] Input the target image into the trained depth estimation model to obtain a relative depth map corresponding to the target image;

[0008] Among them, the depth estimation model consists of a feature extraction module, a plane structure instance segmentation module and a monocular depth estimation module; the depth estimation model is trained according to a multi-task loss function based on contrastive learning and a sample scene image set with depth value labels and plane instance labels.

[0009] Optionally, inputting the target image into a trained depth estimation model to obtain a relative depth map corresponding to the target image specifically includes:

[0010] Passing the target image through a feature extraction module to obtain a multi-scale feature map of the target image;

[0011] Based on the multi-scale feature map, respectively encode to obtain a context feature sequence and a plane boundary position sequence, and decode according to the context feature sequence and the plane boundary position sequence to obtain a plane instance sequence;

[0012] Based on the multi-scale feature map, obtaining an initial monocular depth estimation result of the target image through a depth decoder;

[0013] Input the plane instance sequence into a plane parameter decoder, estimate the plane normal vector of each plane instance, and estimate the three-dimensional coordinates of the plane center point of each plane instance by the multi-dimensional coordinate mean of each pixel in each plane instance;

[0014] Based on the plane normal vector and the three-dimensional coordinates of the plane center point, the initial monocular depth estimation result is corrected to obtain a relative depth map corresponding to the target image.

[0015] Optionally, encoding to obtain a plane boundary position sequence based on the multi-scale feature map specifically includes:

[0016] Detecting a plurality of boundary line segments of the target image based on a Holistically-Attracted Wireframe Parsing (HAWP) algorithm;

[0017] Based on the shallow features in the multi-scale feature map, extract the position code of each plane boundary line segment and the features of each plane boundary line segment, and respectively splice all the plane boundary line segment position codes and all the plane boundary line segment features;

[0018] The plane boundary position sequence is encoded based on the position codes of all the plane boundary line segments after being spliced ​​and the features of all the plane boundary line segments after being spliced.

[0019] Optionally, decoding the plane instance sequence according to the context feature sequence and the plane boundary position sequence specifically includes:

[0020] Inputting the context feature sequence and the plane boundary position sequence into a plane instance decoder in parallel; wherein the plane instance decoder comprises two parallel decoder branches, and the two parallel decoder branches share the same plane query; each of the decoder branches is composed of a plurality of cascaded Transformer decoder layers, and a multi-head self-attention mechanism and a cross-attention mechanism are introduced in each decoder layer;

[0021] The results of the two parallel decoder branches are concatenated and decoded to obtain the plane instance sequence.

[0022] Optionally, the obtaining, based on the multi-scale feature map, an initial monocular depth estimation result of the target image through a depth decoder specifically includes:

[0023] Inputting the multi-scale feature maps into a depth decoder in parallel, fusing the multi-scale depth feature information through a U-shaped network including skip connections, convolutional neural networks, and deconvolution upsampling, to obtain a depth probability distribution map;

[0024] Based on the preset relative depth value prior, each pixel of the depth probability distribution map is mapped to the relative depth value corresponding to the maximum a posteriori probability, so as to obtain the initial monocular depth estimation result of the target image.

[0025] Optionally, the correcting the initial monocular depth estimation result based on the plane normal vector and the three-dimensional coordinates of the plane center point to obtain a relative depth map corresponding to the target image specifically includes:

[0026] For any pixel point belonging to any plane instance, embedding a plane normal vector of the arbitrary plane instance and a three-dimensional coordinate of a plane center point of the arbitrary plane instance into the arbitrary pixel point;

[0027] Determine the theoretical depth value of the arbitrary pixel point according to the plane normal vector embedded in the arbitrary pixel point and the three-dimensional coordinates of the plane center point, as well as the vertical relationship of the vectors; and determine the depth estimation value of the arbitrary pixel point based on the initial monocular depth estimation result;

[0028] When a relative error between the depth estimation value of the arbitrary pixel point and the theoretical depth value is greater than a set threshold, the arbitrary pixel point is corrected based on the average of the depth estimation values ​​of the eight neighborhoods of the arbitrary pixel point to obtain a depth correction value of the arbitrary pixel point;

[0029] Based on the depth estimation value, theoretical depth value and depth correction value of the arbitrary pixel point, a relative depth map corresponding to the target image is determined.

[0030] Optionally, the contrastive learning based multi-task loss function is determined according to a monocular depth estimation loss, a plane instance segmentation loss and a cross-task contrastive loss.

[0031] The present invention also provides a monocular depth estimation device integrating planar structure information, comprising:

[0032] A construction and training unit, used for constructing a depth estimation model, and training the depth estimation model according to a multi-task loss function based on contrastive learning and a set of sample scene images with relative depth value labels and plane instance labels; the depth estimation model consists of a feature extraction module, a plane structure instance segmentation module and a monocular depth estimation module;

[0033] The acquisition unit is used to input the target image into the trained depth estimation model to obtain a relative depth map corresponding to the target image.

[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a monocular depth estimation method for fusing planar structure information as described in any one of the above is implemented.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a monocular depth estimation method for fusing planar structure information as described in any one of the above.

[0036] The monocular depth estimation method and device integrating plane structure information provided by the present invention form a depth estimation model through a feature extraction module, a plane structure instance segmentation module and a monocular depth estimation module, and then obtain a trained depth estimation model according to a multi-task loss function based on contrastive learning and a sample scene image set with relative depth value labels and plane instance labels. The monocular depth estimation task can be assisted by the plane structure information in the scene image, thereby improving the robustness and accuracy of the monocular depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A schematic diagram of the flow of a monocular depth estimation method for fusing planar structure information provided by the present invention;

[0039] Figure 2A schematic diagram of the depth estimation model structure provided by the present invention;

[0040] Figure 3 A schematic diagram of the structure of a context feature encoder provided by the present invention;

[0041] Figure 4 A schematic diagram of the structure of a plane boundary position encoder provided by the present invention;

[0042] Figure 5 A schematic diagram of the structure of a plane example decoder provided by the present invention;

[0043] Figure 6 A schematic diagram of the structure of a depth decoder provided by the present invention;

[0044] Figure 7 A schematic diagram of the structure of a plane parameter decoder provided by the present invention;

[0045] Figure 8 A schematic diagram of the theoretical depth value calculation principle provided by the present invention;

[0046] Fig. 9 A schematic diagram of a flow chart of an embodiment of a monocular depth estimation method for integrating planar structure information provided by the present invention;

[0047] Fig.10 A schematic diagram of the structure of a monocular depth estimation device that integrates planar structure information provided by the present invention;

[0048] Fig.11 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Figure 1 A flow chart of a monocular depth estimation method for integrating planar structure information provided by the present invention is shown in FIG. Figure 1 As shown, the method comprises the following steps:

[0051] Step 101: construct a depth estimation model, and train the depth estimation model according to a multi-task loss function based on contrastive learning and a set of sample scene images with relative depth value labels and plane instance labels; the depth estimation model consists of a feature extraction module, a plane structure instance segmentation module and a monocular depth estimation module.

[0052] Specifically, Figure 2 A schematic diagram of the depth estimation model structure provided by the present invention is shown in FIG. Figure 2 As shown, an embodiment of the present invention provides a depth estimation model consisting of a feature extraction module, a plane structure instance segmentation module and a monocular depth estimation module.

[0053] The feature extraction module can extract the multi-scale feature map of the image, and the extracted multi-scale feature map is used for subsequent processing by the plane structure instance segmentation module and the monocular depth estimation module.

[0054] In some implementations, the feature extraction module can use HRNet-W32 as the backbone network to extract feature maps of four different scales: The height of the image is H. i =H / 2 i , the image width is W i =W / 2 i , C i =W×2 i-4 It indicates the number of channels corresponding to the feature map.

[0055] The plane structure instance segmentation module can extract the plane structure information of the image based on the multi-scale feature map to obtain the plane instance. The monocular depth estimation module can fuse the plane instance to estimate the relative depth of the image pixels, thereby obtaining a relative depth map that fuses the plane structure information.

[0056] The sub-modules for extracting the planar structure contained in the scene image through instance segmentation include: a context feature encoder built on Transformer, a plane boundary position encoder built based on the HAWP line segment detection algorithm and MLP, and a plane instance decoder built based on Transformer.

[0057] Figure 3 A schematic diagram of the structure of the context feature encoder provided by the present invention, such as Figure 3 As shown in the figure, in the context feature encoder, 6 cascaded standard Transformer encoder layers are used to encode the context features contained in the feature map F4 output by the backbone network into a context feature sequence; in each encoder layer, after the three basic parameters K, Q, and V are input, a multi-head attention operation is first performed, and the obtained feature output is added to the input value and then standardized, and then input into the fully connected feedforward neural network to map to the output space. In order to ensure that the standardized numerical output is obtained and the original input is retained, an addition and standardization operation is performed before the output of each layer.

[0058] Figure 4 The schematic diagram of the structure of the plane boundary position encoder provided by the present invention is as follows: Figure 4As shown in the figure, in the plane boundary position encoder, the pre-trained HAWP model is used to detect the plane boundary segments. After n plane boundary segments are detected, the feature vector and position encoding vector of each plane boundary segment at the two end points are sampled by bilinear interpolation, and then an MLP is used to comprehensively extract the endpoint features and position encoding of the corresponding segment. Finally, the endpoint features and position encoding sequences of all n plane boundary segments are simultaneously input into another MLP to be encoded into the final plane boundary position sequence.

[0059] Figure 5 A schematic diagram of a planar example decoder structure provided by the present invention, such as Figure 5 As shown in the figure, in the plane instance decoder, the context feature sequence and the plane boundary position sequence are used as parallel branch inputs and share the same plane query. In the two branches of the decoder, 6 cascaded decoder layers are set. In addition to multi-head attention operations, addition and normalization operations, and feedforward neural networks, cross-attention operations are also introduced to achieve interaction between the context feature sequence and the plane boundary position sequence, so that the shared plane query can consider the context features and plane boundary positions of the scene image as a whole to perceive the plane instance, and finally splice the results of the two branches to decode the plane instance sequence.

[0060] Step 102: Input the target image into the trained depth estimation model to obtain a relative depth map corresponding to the target image.

[0061] The target image refers to the image for which monocular depth estimation is required. After the trained depth estimation model is obtained according to the above loss function and the sample scene image set, the target image can be input into the trained depth estimation model to obtain the relative depth map corresponding to the target image.

[0062] Input the target image into the trained depth estimation model to obtain the relative depth map corresponding to the target image, including:

[0063] Pass the target image through the feature extraction module to obtain a multi-scale feature map of the target image;

[0064] Based on the multi-scale feature map, the context feature sequence and the plane boundary position sequence are encoded respectively, and the plane instance sequence is decoded according to the context feature sequence and the plane boundary position sequence;

[0065] Based on the multi-scale feature map, the initial monocular depth estimation result of the target image is obtained through the depth decoder;

[0066] Input the plane instance sequence into the plane parameter decoder to estimate the plane normal vector and the three-dimensional coordinates of the plane center point of each plane instance;

[0067] Based on the plane normal vector and the three-dimensional coordinates of the plane center point, the initial monocular depth estimation result is corrected to obtain the relative depth map corresponding to the target image.

[0068] Specifically, in the process of obtaining the depth map of the target image through the trained depth estimation model, the target image needs to be first passed through the feature extraction module to obtain a multi-scale feature map of the target image. For example, using HRNet-W32 as the backbone network, four feature maps of different scales are extracted, namely, high-resolution, higher-resolution, lower-resolution and low-resolution feature maps.

[0069] After obtaining the multi-scale feature map, the plane structure information of the target image can be extracted through the plane structure instance segmentation module. In this process, the multi-scale feature map can be passed through the context feature encoder and the plane boundary position encoder in the plane structure instance segmentation module respectively to obtain the context feature sequence and the plane boundary position sequence.

[0070] In some embodiments, the context feature encoder in the embodiments of the present invention can be built based on Transformer, for example, using a 6-layer cascaded standard Transformer multi-head self-attention encoder, which can be a 16-head self-attention encoder, to encode the context features in the last layer of feature maps extracted by the backbone network into a context feature sequence.

[0071] In some implementation manners, the plane boundary position encoder in the embodiment of the present invention can be constructed by using the HAWP algorithm and a multilayer perceptron (MLP).

[0072] After obtaining the context feature sequence and the plane boundary position sequence, the two can be input into the plane instance decoder in parallel to decode the plane instance sequence. For example, the plane instance decoder can use a parallel multi-layer cascaded Transformer decoder.

[0073] After obtaining the multi-scale feature map and plane instances, the plane instances can be fused through the monocular depth estimation module to estimate the pixel relative depth, thereby obtaining a relative depth map that integrates the plane structure information.

[0074] Specifically, the depth decoder in the monocular depth estimation module can be used to obtain an initial monocular depth estimation result of the target image based on the depth value prior.

[0075] After obtaining the initial monocular depth estimation result, the plane instance sequence and the depth estimation value are input into the plane parameter decoder in the monocular depth estimation module to estimate the plane normal vector of each plane instance and the three-dimensional coordinates of the plane center point of each plane instance.

[0076] After obtaining the plane parameters and the initial monocular depth estimation results, the theoretical depth value and depth correction value are calculated by combining the two to obtain the final depth value and output the final relative depth map.

[0077] Figure 6 A schematic diagram of the structure of the depth decoder provided by the present invention is shown in FIG. Figure 6 As shown, in the deep decoder, the input is a parallel feature map set Through a U-shaped network structure including skip connections, CNN, and deconvolution upsampling, starting from the smallest scale feature map F4, its deep feature information is decoded by CNN, and the decoding result is restored to the scale corresponding to F3 through deconvolution upsampling. At the same time, the high-level information decoded in F3 and the low-level information decoded in F4 are fused channel by channel through skip connections, and so on, until the original input image scale is restored. The fused multi-scale deep feature information is then transformed into a probability distribution pixel by pixel through the Softmax function to obtain the depth value probability distribution map D′.

[0078] Then, combined with the preset relative depth value prior, D′ is mapped pixel by pixel to the relative depth value corresponding to the maximum posterior probability, that is, a mapping relationship from pixel to depth is constructed to obtain a monocular depth estimation result without fine-tuning.

[0079] Figure 7 A schematic diagram of the plane parameter decoder structure provided by the present invention, such as Figure 7 As shown, the plane instance segmentation result is input into the plane parameter decoder, and the normal vector of each plane instance is estimated using principal component analysis (PCA) combined with the depth estimation value of each pixel within the range of each plane instance.

[0080] Then, for each plane instance, the mean values ​​of the three dimensions of the pixel coordinates (x, y, d) belonging to the plane are calculated respectively as the three-dimensional coordinate estimation value of the center point of the plane instance.

[0081] According to the plane normal vector and the three-dimensional coordinates of the plane center point of each plane instance, the theoretical depth value and depth correction value of each pixel point in the target image can be calculated respectively, so as to correct the initial monocular depth estimation result of the target image and obtain the relative depth map corresponding to the target image.

[0082] In some implementations, for each plane instance, the plane parameter decoder may use PCA to obtain the eigenvector corresponding to the minimum eigenvalue of the coordinate covariance matrix of each point on the plane to estimate the plane normal vector.

[0083] For example, the plane instance sequence S p=(x, y, cls) Input plane parameter decoder, where (x, y) is the coordinate of the pixel in the camera coordinate system, cls represents the plane instance label to which it belongs (cls = -1 means that the pixel is a free point that does not belong to any plane instance), combined with the depth estimation value d of each pixel within the range of each plane instance, PCA is used to estimate its normal vector for each plane instance: First, the three-dimensional coordinates (x, y, d) of each pixel are combined into a matrix (n represents the number of pixels contained in the plane instance), then subtract the row mean from each row of A to achieve zero mean to obtain A′, then calculate the covariance matrix Cov(A′) of A′, and finally obtain the unit eigenvector corresponding to the minimum eigenvalue of Cov(A′) as the normal vector estimate of the plane instance.

[0084] Figure 8 A schematic diagram of the theoretical depth value calculation principle provided by the present invention, such as Figure 8 As shown, let the actual coordinate of pixel point p after depth estimation be (x p ,y p , d), the point is the theoretical point p in the plane t The coordinates of The plane normal vector embedded in point p and the coordinates of the center point c of the plane (c x , c y , c z ), by the perpendicular relationship between the vectors Uniquely determine its theoretical depth value d t :

[0085]

[0086] If the absolute error between its depth estimation value and its theoretical depth value is greater than a set threshold, the average of the depth estimation values ​​of the eight neighborhoods of point p can be used as its depth correction value.

[0087] For example, if the depth estimation value d is equal to the theoretical value d t If the relative error is greater than 10%, the depth estimation value of the eight neighborhoods N8(p) of point p is corrected to determine the depth estimation correction value d′ (for pixels at the edge of the image, the default value in the eight neighborhoods is supplemented with the mean of the theoretical depth value of the center pixel and the actual estimated value):

[0088]

[0089] In some implementations, a weighted average of the depth estimation value, theoretical depth value, and depth correction value of each pixel may be calculated to obtain a final depth map.

[0090] Preferably, the formula for determining the final depth value of point p is as follows:

[0091]

[0092] It should be noted that the plane instance label (cls) is used to refer to two situations in the embodiment of the present invention. For each pixel whose plane instance label is not -1, it indicates that the pixel belongs to a certain plane instance, and the two parameters of the normal vector and the center point coordinates of the plane instance to which it belongs can be embedded. For each pixel whose instance label is -1, it indicates that the pixel does not belong to any plane instance, and the mean of the depth estimation values ​​of the eight neighborhoods can be directly used to adjust it.

[0093] Fig. 9 The flowchart of the embodiment of the monocular depth estimation method for integrating planar structure information provided by the present invention is as follows: Fig. 9 As shown, the specific steps of this method are as follows:

[0094] S1. Based on the HRNet-W32 backbone network to extract multi-scale feature maps, the plane structure information is extracted based on instance segmentation, and the global context information is encoded using a multi-layer Transformer encoder with a dynamic receptive field to obtain a context feature sequence. At the same time, an encoder composed of line segment detection and MLP is used in parallel to encode the features and positions of the plane boundary segments to obtain a plane boundary position sequence. These two sequences are then input in parallel into a multi-layer Transformer decoder with a cross-attention mechanism to decode the plane instance sequence.

[0095] Optionally, based on the multi-scale feature map, encoding is performed to obtain a plane boundary position sequence, specifically including:

[0096] Detecting a plurality of boundary line segments of the target image based on an overall attracting wireframe parsing algorithm;

[0097] Based on the shallow features in the multi-scale feature map, extract the position code of each plane boundary line segment and the features of each plane boundary line segment, and respectively splice all the plane boundary line segment position codes and all the plane boundary line segment features;

[0098] The plane boundary position sequence is encoded based on the position codes of all the plane boundary line segments after being spliced ​​and the features of all the plane boundary line segments after being spliced.

[0099] Specifically, in the process of obtaining a plane boundary position sequence through multi-scale feature map encoding, a plane boundary position encoder can be built based on the HAWP algorithm and MLP. First, the HAWP algorithm is used to detect multiple plane boundary line segments of the target image.

[0100] After detecting multiple plane boundary line segments, the plane boundary position encoder can extract the position code of each plane boundary line segment and the features of each plane boundary line segment, and respectively splice the position codes of all plane boundary line segments and the features of all plane boundary line segments, and encode the plane boundary position sequence according to the spliced ​​position codes of all plane boundary line segments and the spliced ​​features of all plane boundary line segments.

[0101] In some implementations, the feature vectors (f1, f2) and position encoding vectors (e1, e2) of each plane boundary line segment at both end points can be sampled by bilinear interpolation, and then an MLP can be used to comprehensively extract the feature f of the corresponding line segment l. l and its position code l :

[0102]

[0103] In the formula, Represents the vector concatenation operation. Finally, the features and position codes of all the plane boundary segments are concatenated into S′ l 、E l , and simultaneously input another MLP to output the final plane boundary position sequence S l .

[0104] Optionally, decoding the context feature sequence and the plane boundary position sequence to obtain a plane instance sequence specifically includes:

[0105] The context feature sequence and the plane boundary position sequence are input into the plane instance decoder in parallel; the plane instance decoder includes two parallel decoder branches, and the two parallel decoder branches share the same plane query; each decoder branch is composed of multiple cascaded Transformer decoder layers, and a multi-head self-attention mechanism and a cross-attention mechanism are introduced in each decoder layer; the results of the two parallel decoder branches are spliced ​​and decoded to obtain a plane instance sequence.

[0106] Specifically, the plane instance decoder in the embodiment of the present invention can be built based on Transformer, which includes two parallel decoder branches, and the two parallel decoder branches share the same plane query. For each decoder branch, it can be composed of multiple cascaded Transformer decoder layers, and a multi-head self-attention mechanism and a cross-attention mechanism are introduced in each decoder layer.

[0107] In some embodiments, with the context feature sequence and the plane boundary position sequence as parallel inputs, the two parallel decoder branches can be based on the standard Transformer decoder, which is also composed of 6 cascaded decoder layers. In addition to the multi-head self-attention mechanism, each decoder layer also introduces a cross-attention mechanism to achieve interaction between the context feature sequence and the plane boundary position sequence, so that plane queries can consider the image context features and plane boundary positions as a whole to perceive plane instances.

[0108] Finally, the results of the two parallel decoder branches are concatenated and decoded to obtain a sequence of plane instances.

[0109] S2. Input the feature maps of each scale into the depth decoder in parallel, estimate the probability distribution of the depth value of each pixel through the U-type network, combine the preset relative depth value prior, build the pixel-to-depth mapping relationship according to the relative depth value corresponding to the maximum posterior probability, and gradually upsample to restore it to the original input image scale.

[0110] like Figure 2 As shown, in step S2, each scale feature map is input into the depth decoder in parallel to construct a mapping from pixel to relative depth, and obtain an unfine-tuned pixel depth estimation value.

[0111] Optionally, based on the multi-scale feature map, an initial monocular depth estimation result of the target image is obtained through a depth decoder, specifically including:

[0112] The multi-scale feature maps are input into the depth decoder in parallel, and the multi-scale depth feature information is fused through a U-shaped network including skip connection, convolutional neural network and deconvolution upsampling to obtain a relative depth probability distribution map;

[0113] Based on the preset relative depth value prior, each pixel of the depth probability distribution map is mapped to the relative depth value corresponding to the maximum posterior probability, and the initial monocular depth estimation result of the target image is obtained.

[0114] Specifically, in the process of obtaining the initial monocular depth estimation result of the target image through the depth decoder, the feature maps of each scale can be input into the depth decoder in parallel, and the relative depth value probability distribution of each pixel can be estimated through the U-type network structure. The U-type structure includes skip connections, convolutional neural networks, and deconvolution upsampling.

[0115] The following uses four scale feature maps as an example to illustrate the process. The depth decoder is input in parallel, starting from the smallest-scale feature map F4 through a U-shaped structure including skip connections, CNN, and deconvolution upsampling. The two-layer CNN is used to decode its depth feature information, and the decoded result is restored to the scale corresponding to F3 through deconvolution upsampling. At the same time, the high-level information in F3 is fused with the low-level information in F4 on the channel through skip connections, and so on, until it is restored to the same scale as the original input image and the depth probability distribution map of multi-scale depth feature information is fused (the depth value probability distribution can be estimated pixel by pixel by the Softmax function).

[0116] After obtaining the depth probability distribution map, it can be combined with a preset relative depth value prior (in some implementations, the relative depth value range can be set to 0 to 63, with a total of 64 levels of relative depth), and a pixel-to-depth mapping relationship can be constructed based on the relative depth value corresponding to the maximum a posteriori probability, that is, each pixel is mapped to the relative depth value corresponding to the maximum a posteriori probability, and gradually upsampled to restore it to the original input image scale to obtain the initial monocular depth estimation result of the target image.

[0117] S3. Integrate the plane structure information in the monocular depth estimation network, input the plane instance sequence into the plane parameter decoder, and obtain two plane parameters for each plane instance, including the plane normal vector and the three-dimensional coordinates of the plane center point.

[0118] like Figure 2 As shown, in step S3, the plane instance segmentation result is input into the plane parameter decoder, and combined with the pixel depth value estimated in step S2, two parameters, the plane normal vector and the three-dimensional coordinates of the plane center point, are estimated for each plane instance.

[0119] Optionally, based on the plane instance sequence and the initial monocular depth estimation result, a normal vector and a three-dimensional coordinate of a center point of each plane instance are obtained through a plane parameter decoder, specifically including:

[0120] Use PCA to obtain the eigenvector corresponding to the minimum eigenvalue of the covariance matrix of the coordinates of each point on the plane to estimate the plane normal vector;

[0121] For each plane instance, the mean value of the coordinates of each pixel belonging to the plane is calculated as the three-dimensional coordinate estimate of the center point of the plane instance.

[0122] Specifically, the plane instance sequence S p=(x, y, cls) Input plane parameter decoder, where (x, y) is the coordinate of the pixel in the camera coordinate system, cls represents the plane instance label to which it belongs (cls = -1 means that the pixel is a free point that does not belong to any plane instance), combined with the depth estimation value d of each pixel within the range of each plane instance, PCA is used to estimate its normal vector for each plane instance: First, the three-dimensional coordinates (x, y, d) of each pixel are combined into a matrix (n represents the number of pixels contained in the plane instance), then subtract the row mean from each row of A to achieve zero mean to obtain A′, then calculate the covariance matrix Cov(A′) of A′, and finally obtain the unit eigenvector corresponding to the minimum eigenvalue of Cov(A′) as the normal vector estimate of the plane instance.

[0123] S4. Fine-tune the depth estimation value based on the plane parameter information. For pixels belonging to the plane instance, the plane normal vector and the plane center point coordinates are embedded pixel by pixel. The theoretical depth value of the pixel is calculated based on the depth relationship provided by the normal vector and the center point coordinates. If the absolute error between the estimated value and the theoretical value is greater than the set threshold, the depth estimation mean of the 8-neighborhood of the pixel is used as the correction value. Finally, the weighted average of the relative depth estimation value, theoretical value and correction value of each pixel is calculated to map the final depth map. For free pixels, the average of the 8-neighborhood pixel depth estimation is directly used for fine-tuning.

[0124] like Figure 2 As shown, in step S4, the depth estimation value is fine-tuned based on the plane parameter information, the plane parameters are pixel-level associated and embedded, and the depth estimation value is fine-tuned pixel by pixel with the assistance of the two parameters of the normal vector and the center point coordinates to map the final relative depth map.

[0125] Optionally, based on the plane normal vector and the three-dimensional coordinates of the plane center point, the initial monocular depth estimation result is corrected to obtain a relative depth map corresponding to the target image, specifically including:

[0126] For any pixel point belonging to any plane instance, the plane normal vector of the plane instance and the three-dimensional coordinates of the plane center point of the plane instance are embedded into the arbitrary pixel point;

[0127] Determine the theoretical depth value of any pixel point based on the plane normal vector embedded in any pixel point, the three-dimensional coordinates of the plane center point, and the vertical relationship between the vectors; and determine the depth estimation value of any pixel point based on the initial monocular depth estimation result;

[0128] When the relative error between the depth estimation value and the theoretical depth value of any pixel point is greater than a set threshold, the arbitrary pixel point is corrected based on the average of the depth estimation values ​​of the eight neighborhoods of the arbitrary pixel point to obtain a depth correction value of the arbitrary pixel point;

[0129] Based on the depth estimation value, theoretical depth value and depth correction value of any pixel point, the relative depth map corresponding to the target image is determined.

[0130] Specifically, in the process of correcting the initial monocular depth estimation result based on the plane normal vector and the three-dimensional coordinates of the plane center point, the two parameters of the plane normal vector and the three-dimensional coordinates of the plane center point can firstly be associated and embedded pixel by pixel.

[0131] Then, the theoretical depth value of each pixel can be calculated based on the plane normal vector and the three-dimensional coordinates of the plane center point, as well as the vertical relationship between the vectors, and the depth estimation value of each pixel can be determined based on the initial monocular depth estimation result.

[0132] For example, suppose the actual coordinates of pixel point p in the camera coordinate system after depth estimation are (x p ,y p , d), the point is the theoretical point p in the plane t The coordinates of The plane normal vector embedded in point p and the coordinates of the center point c of the plane (c x , c y , c z ), by the perpendicular relationship between the vectors Uniquely determine its theoretical depth value d t :

[0133]

[0134] If the depth estimation value d is equal to the theoretical value d t If the relative error is greater than 10%, the depth estimation value of the eight neighborhoods N8(p) of point p is corrected to determine the depth estimation correction value d′ (for pixels at the edge of the image, the default value in the eight neighborhoods is supplemented with the mean of the theoretical depth value of the center pixel and the actual estimated value):

[0135]

[0136] Finally, the weighted average of the depth estimation value, theoretical depth value and depth correction value of each pixel is calculated to determine the final depth value of point p:

[0137]

[0138] It should be noted that the plane instance label (cls) in the above formula refers to two cases. For each pixel whose plane instance label is not -1, it means that the pixel belongs to a certain plane instance, and the two parameters of the normal vector and the center point coordinates of the plane instance to which it belongs can be embedded. For each pixel whose instance label is -1, it means that the pixel does not belong to any plane instance, and it can be directly adjusted by the mean of the depth estimation values ​​of the eight neighborhoods.

[0139] Optionally, the depth estimation model can be obtained through supervised training based on a multi-task loss function based on contrastive learning and a set of sample scene images with both relative depth value labels and plane instance labels.

[0140] A high-quality indoor scene RGB image dataset that meets the requirements is selected, each image is cropped to a pixel size of 256×192, and a multi-task loss function L based on contrastive learning is designed to train the depth estimation model provided in this embodiment, where L is composed of the monocular depth estimation loss L mde , plane instance segmentation loss L seg and cross-task contrast loss L ctc It consists of three parts and adopts an uncertain weight strategy:

[0141]

[0142] Preferably, the multi-task loss function based on contrastive learning in the embodiment of the present invention can be determined according to the monocular depth estimation loss, the plane instance segmentation loss and the cross-task contrast loss.

[0143] In some implementations, the loss function L may be determined according to the following formula:

[0144]

[0145] Among them, L mde represents the monocular depth estimation loss, L seg represents the plane instance segmentation loss, L ctc represents the cross-task contrast loss. S1 and S2 are both non-negative learnable parameters. The uncertain weight strategy allows the model to automatically adjust the relative importance of the two tasks. α is a model hyperparameter, which can be set to 0 in the first 50% of the iterations of the training process and to 0.1 in the next 50% of the iterations.

[0146] Monocular depth estimation loss L mde The calculation is as follows:

[0147]

[0148] Among them, j represents the number of the pixel corresponding to the depth estimation value, and Represent the final depth estimation value and the true depth value (i.e., the relative depth value label), respectively. It means to calculate the mean absolute error.

[0149] Plane instance segmentation loss L seg Defined as the predicted plane instance s p and the ground-truth plane instance (i.e., plane instance label) The arithmetic mean of the negative logarithms of the intersection ratios is calculated as follows:

[0150]

[0151] Among them, m represents the total number of plane instances, i represents the plane instance number, IoU represents the intersection over union ratio, and area represents the number of pixels contained in the shown area, that is, the discrete area.

[0152] Cross-task contrast loss L ctc The aim is to maximize the pixel-level similarity between the corresponding plane instance segmentation results and the depth estimation map through contrastive learning method. The calculation method is as follows:

[0153]

[0154] Where N represents the number of original images input in a single batch of model training. and The calculation method is as follows:

[0155]

[0156]

[0157] in, It represents the cosine similarity between the pixel-by-pixel features of the i-th pair of plane instance segmentation results and the depth estimation map. t represents the temperature coefficient in contrastive learning, which is used to adjust the similarity measurement results between samples. It is usually a small positive number and can be set to 0.1 here.

[0158] After training, you can load the model parameters learned in training based on the model, input a single indoor scene image, and estimate the relative depth of the scene through the two-stage tasks of plane instance segmentation and monocular depth estimation, and output a relative depth map.

[0159] The monocular depth estimation method provided in this embodiment not only improves the robustness and accuracy of monocular depth estimation of indoor scenes by extracting and fusing planar structure information in indoor scene images, but also explores a reliable and effective practical method for the monocular depth estimation technology route of "fusing structural information".

[0160] The monocular depth estimation device for fusing plane structure information provided by the present invention is described below. The monocular depth estimation device for fusing plane structure information described below and the monocular depth estimation method for fusing plane structure information described above can correspond to each other.

[0161] Fig.10 The schematic diagram of the structure of the monocular depth estimation device integrating the planar structure information provided by the present invention is as follows: Fig.10 As shown, the device comprises:

[0162] A construction and training unit 1001 is used to construct a depth estimation model and train the depth estimation model according to a multi-task loss function based on contrastive learning and a sample scene image set with relative depth value labels and plane instance labels; the depth estimation model consists of a feature extraction module, a plane structure instance segmentation module and a monocular depth estimation module;

[0163] The acquisition unit 1002 is used to input the target image into the trained depth estimation model to obtain a relative depth map corresponding to the target image.

[0164] Optionally, the target image is input into a trained depth estimation model to obtain a relative depth map corresponding to the target image, specifically including:

[0165] Pass the target image through the feature extraction module to obtain a multi-scale feature map of the target image;

[0166] Based on the multi-scale feature map, the context feature sequence and the plane boundary position sequence are encoded respectively, and the plane instance sequence is decoded according to the context feature sequence and the plane boundary position sequence;

[0167] Based on the multi-scale feature map, the initial monocular depth estimation result of the target image is obtained through the depth decoder;

[0168] Input the plane instance sequence into the plane parameter decoder, estimate the plane normal vector of each plane instance, and estimate the three-dimensional coordinates of the plane center point of each plane instance by the mean of the three-dimensional coordinates of each pixel in each plane example;

[0169] Based on the plane normal vector and the three-dimensional coordinates of the plane center point, the initial monocular depth estimation result is corrected to obtain the relative depth map corresponding to the target image.

[0170] Optionally, based on the multi-scale feature map, encoding is performed to obtain a plane boundary position sequence, specifically including:

[0171] Detecting a plurality of boundary line segments of the target image based on an overall attracting wireframe parsing algorithm;

[0172] Based on the shallow features in the multi-scale feature map, extract the position code of each plane boundary line segment and the features of each plane boundary line segment, and respectively splice all the plane boundary line segment position codes and all the plane boundary line segment features;

[0173] The plane boundary position sequence is encoded based on the position codes of all the plane boundary line segments after being spliced ​​and the features of all the plane boundary line segments after being spliced.

[0174] Optionally, decoding the context feature sequence and the plane boundary position sequence to obtain a plane instance sequence specifically includes:

[0175] The context feature sequence and the plane boundary position sequence are input into the plane instance decoder in parallel; the plane instance decoder includes two parallel decoder branches, and the two parallel decoder branches share the same plane query; each decoder branch is composed of multiple cascaded Transformer decoder layers, and a multi-head self-attention mechanism and a cross-attention mechanism are introduced in each decoder layer;

[0176] The results of the two parallel decoder branches are concatenated and decoded to obtain a sequence of plane instances.

[0177] Optionally, based on the multi-scale feature map, an initial monocular depth estimation result of the target image is obtained through a depth decoder, specifically including:

[0178] The multi-scale feature maps are input into the deep decoder in parallel, and the multi-scale deep feature information is fused through a U-shaped network including skip connection, convolutional neural network and deconvolution upsampling to obtain a deep probability distribution map;

[0179] Based on the preset relative depth value prior, each pixel of the depth probability distribution map is mapped to the relative depth value corresponding to the maximum posterior probability, and the initial monocular depth estimation result of the target image is obtained.

[0180] Optionally, based on the plane normal vector and the three-dimensional coordinates of the plane center point, the initial monocular depth estimation result is corrected to obtain a relative depth map corresponding to the target image, specifically including:

[0181] For any pixel point belonging to any plane instance, the plane normal vector of the plane instance and the three-dimensional coordinates of the plane center point of the plane instance are embedded into the arbitrary pixel point;

[0182] Determine the theoretical depth value of any pixel point based on the plane normal vector embedded in any pixel point, the three-dimensional coordinates of the plane center point, and the vertical relationship between the vectors; and determine the depth estimation value of any pixel point based on the initial monocular depth estimation result;

[0183] When the relative error between the depth estimation value and the theoretical depth value of any pixel point is greater than a set threshold, the arbitrary pixel point is corrected based on the average of the depth estimation values ​​of the eight neighborhoods of the arbitrary pixel point to obtain a depth correction value of the arbitrary pixel point;

[0184] Based on the depth estimation value, theoretical depth value and depth correction value of any pixel point, the relative depth map corresponding to the target image is determined.

[0185] Optionally, a multi-task loss function based on contrastive learning is determined by supervising the depth estimation model according to a monocular depth estimation loss, a plane instance segmentation loss, and a cross-task contrastive loss.

[0186] Fig.11 A schematic diagram of the structure of an electronic device provided by the present invention, such as Fig.11 As shown, the electronic device may include: a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 may call the logic instructions in the memory 1130 to execute the monocular depth estimation method for fusing planar structure information provided by the above methods, and the method includes:

[0187] Constructing a depth estimation model, and training the depth estimation model according to a multi-task loss function based on contrastive learning and a set of sample scene images with relative depth value labels and plane instance labels; the depth estimation model consists of a feature extraction module, a plane structure instance segmentation module, and a monocular depth estimation module;

[0188] The target image is input into the trained depth estimation model to obtain a relative depth map corresponding to the target image.

[0189] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0190] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the monocular depth estimation method for fusing planar structure information provided by the above methods, the method comprising:

[0191] Constructing a depth estimation model, and training the depth estimation model according to a multi-task loss function based on contrastive learning and a set of sample scene images with relative depth value labels and plane instance labels; the depth estimation model consists of a feature extraction module, a plane structure instance segmentation module, and a monocular depth estimation module;

[0192] The target image is input into the trained depth estimation model to obtain a relative depth map corresponding to the target image.

[0193] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0194] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A monocular depth estimation method integrating planar structure information, characterized in that: include: Constructing a depth estimation model, and training the depth estimation model according to a multi-task loss function based on contrastive learning and a set of sample scene images with relative depth value labels and plane instance labels; the depth estimation model consists of a feature extraction module, a plane structure instance segmentation module, and a monocular depth estimation module; Input the target image into the trained depth estimation model to obtain a relative depth map corresponding to the target image; Inputting the target image into a trained depth estimation model to obtain a relative depth map corresponding to the target image specifically includes: Passing the target image through a feature extraction module to obtain a multi-scale feature map of the target image; Based on the multi-scale feature map, respectively encode to obtain a context feature sequence and a plane boundary position sequence, and decode according to the context feature sequence and the plane boundary position sequence to obtain a plane instance sequence; Based on the multi-scale feature map, obtaining an initial monocular depth estimation result of the target image through a depth decoder; Input the plane instance sequence into a plane parameter decoder, estimate the plane normal vector of each plane instance, and estimate the three-dimensional coordinates of the plane center point of each plane instance by the multi-dimensional coordinate mean of each pixel in each plane instance; Based on the plane normal vector and the three-dimensional coordinates of the plane center point, the initial monocular depth estimation result is corrected to obtain a relative depth map corresponding to the target image.

2. The monocular depth estimation method integrating planar structure information according to claim 1, characterized in that: Based on the multi-scale feature map, encoding is performed to obtain a plane boundary position sequence, specifically including: Detecting a plurality of boundary line segments of the target image based on an overall attracting wireframe parsing algorithm; Based on the shallow features in the multi-scale feature map, extract the position code of each plane boundary line segment and the features of each plane boundary line segment, and respectively splice all the plane boundary line segment position codes and all the plane boundary line segment features; The plane boundary position sequence is encoded based on the position codes of all the plane boundary line segments after being spliced ​​and the features of all the plane boundary line segments after being spliced.

3. The monocular depth estimation method integrating planar structure information according to claim 1 or 2, characterized in that: The decoding according to the context feature sequence and the plane boundary position sequence to obtain a plane instance sequence specifically includes: Inputting the context feature sequence and the plane boundary position sequence into a plane instance decoder in parallel; wherein the plane instance decoder comprises two parallel decoder branches, and the two parallel decoder branches share the same plane query; each of the decoder branches is composed of a plurality of cascaded Transformer decoder layers, and a multi-head self-attention mechanism and a cross-attention mechanism are introduced in each decoder layer; The results of the two parallel decoder branches are concatenated and decoded to obtain the plane instance sequence.

4. The monocular depth estimation method integrating planar structure information according to claim 1, characterized in that: The step of obtaining an initial monocular depth estimation result of the target image through a depth decoder based on the multi-scale feature map specifically includes: Inputting the multi-scale feature maps into a depth decoder in parallel, fusing the multi-scale depth feature information through a U-shaped network including skip connections, convolutional neural networks, and deconvolution upsampling, to obtain a depth probability distribution map; Based on the preset relative depth value prior, each pixel of the depth probability distribution map is mapped to the relative depth value corresponding to the maximum a posteriori probability, so as to obtain the initial monocular depth estimation result of the target image.

5. The monocular depth estimation method integrating planar structure information according to claim 1, characterized in that: The correcting the initial monocular depth estimation result based on the plane normal vector and the three-dimensional coordinates of the plane center point to obtain the relative depth map corresponding to the target image specifically includes: For any pixel point belonging to any plane instance, embedding a plane normal vector of the arbitrary plane instance and a three-dimensional coordinate of a plane center point of the arbitrary plane instance into the arbitrary pixel point; Determine the theoretical depth value of the arbitrary pixel point according to the plane normal vector embedded in the arbitrary pixel point and the three-dimensional coordinates of the plane center point, as well as the vertical relationship of the vectors; and determine the depth estimation value of the arbitrary pixel point based on the initial monocular depth estimation result; When a relative error between the depth estimation value of the arbitrary pixel point and the theoretical depth value is greater than a set threshold, the arbitrary pixel point is corrected based on the average of the depth estimation values ​​of the eight neighborhoods of the arbitrary pixel point to obtain a depth correction value of the arbitrary pixel point; Based on the depth estimation value, theoretical depth value and depth correction value of the arbitrary pixel point, a relative depth map corresponding to the target image is determined.

6. The monocular depth estimation method integrating planar structure information according to claim 1, characterized in that: The multi-task loss function based on contrastive learning is determined according to monocular depth estimation loss, plane instance segmentation loss and cross-task contrast loss.

7. A monocular depth estimation device integrating planar structure information, characterized in that: include: A construction and training unit, used for constructing a depth estimation model, and training the depth estimation model according to a multi-task loss function based on contrastive learning and a set of sample scene images with relative depth value labels and plane instance labels; the depth estimation model consists of a feature extraction module, a plane structure instance segmentation module and a monocular depth estimation module; An acquisition unit, used to input a target image into a trained depth estimation model to obtain a relative depth map corresponding to the target image; Inputting the target image into a trained depth estimation model to obtain a relative depth map corresponding to the target image specifically includes: Passing the target image through a feature extraction module to obtain a multi-scale feature map of the target image; Based on the multi-scale feature map, respectively encode to obtain a context feature sequence and a plane boundary position sequence, and decode according to the context feature sequence and the plane boundary position sequence to obtain a plane instance sequence; Based on the multi-scale feature map, obtaining an initial monocular depth estimation result of the target image through a depth decoder; Input the plane instance sequence into a plane parameter decoder, estimate the plane normal vector of each plane instance, and estimate the three-dimensional coordinates of the plane center point of each plane instance by the multi-dimensional coordinate mean of each pixel in each plane instance; Based on the plane normal vector and the three-dimensional coordinates of the plane center point, the initial monocular depth estimation result is corrected to obtain a relative depth map corresponding to the target image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the monocular depth estimation method for fusing planar structure information as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the monocular depth estimation method for fusing planar structure information as described in any one of claims 1 to 6 is implemented.

Images