Method and apparatus for performing planar detection

Abstract

A method and apparatus for performing plane detection are disclosed, the method including: obtaining an input image; extracting features of the input image and estimating a depth map of the input image based on the extracted features using a deep neural network; detecting a planar region in the input image using region segmentation with the depth map.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of human-computer interaction, and more particularly, to a method and apparatus for performing plane detection. BACKGROUND

[0002] In the field of human-computer interaction, augmented reality (AR) is one of the important branches. Augmented reality is an interactive experience of a real-world environment where the objects residing in the real world are augmented with computer-generated sensory information, sometimes spanning multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory. An augmented reality system can be defined as a system that satisfies three basic characteristics: the combination of real and virtual worlds, real-time interaction, and accurate three-dimensional registration of virtual and real objects. The superimposed sensory information can be an addition to the real environment or a masking of the real environment.

[0003] With the popularity of mobile intelligent devices and the tremendous improvement of computing power, augmented reality technology has made great progress in the past few years. As a new human-computer interaction technology, augmented reality can more intuitively display physical objects and data information in the real scene, and more and more researches have begun to explore how to better combine virtual objects with the real environment to provide better immersive sensory experience.

[0004] Plane detection technology is one of the key technologies of augmented reality. This technology can detect the position and size of various planes (such as the ground, the desktop, the wall, etc.) in the real environment. In augmented reality, virtual items can be placed on these detected planes. Accurate plane detection results are one of the key factors for augmented reality applications to provide a good user experience. However, existing plane detection methods not only cannot detect planes in textureless regions (such as walls, desktops, etc.) in the image scene, but also cannot obtain planes that align with the boundaries of real objects. In view of this, there is a need for a method and apparatus that can accurately detect planes. SUMMARY

[0005] According to an aspect of the present disclosure, a method for performing plane detection is provided, comprising: obtaining an input image; extracting features of the input image and estimating a depth map of the input image based on the extracted features using a deep neural network; and detecting a plane region in the input image using region segmentation with the depth map.

[0006] According to an example embodiment, the deep neural network can include a feature extractor for extracting features of an input image, a depth estimation branch for estimating depth information of the input image, and a normal estimation branch for estimating normal information of the input image, wherein the normal information estimated by the normal estimation branch is used to optimize the depth information estimated by the depth estimation branch in estimating a depth map of the input image.

[0007] According to an example embodiment, in estimating a depth map of an input image, a feature map of a predetermined resolution obtained by extracting features of the input image using a feature extractor can be fused with a depth feature map of the same resolution generated in a depth estimation process using a depth estimation branch and a normal feature map of the same resolution generated in a normal estimation process using a normal estimation branch, to obtain the depth map using the fused depth feature map and the fused normal feature map.

[0008] According to an example embodiment, the optimization of the depth information estimated by the depth estimation branch using the normal information estimated by the normal estimation branch can include extracting information about a region in which a normal feature changes by more than a predetermined degree in the normal feature map and optimizing the depth feature map using the information to obtain an optimized depth feature map.

[0009] According to an example embodiment, the extraction of the information about the region in which the normal feature changes by more than the predetermined degree in the normal feature map and the optimization of the depth feature map using the information can include performing horizontal and vertical depth convolutions on the normal feature map, respectively, and obtaining a horizontal attention map and a vertical attention map for the information using an activation function, and obtaining the optimized depth feature map based on the horizontal attention map, the vertical attention map, and the depth feature map.

[0010] According to an example embodiment, the obtaining of the optimized depth feature map based on the horizontal attention map, the vertical attention map, and the depth feature map can include performing a weighting process on the horizontal and vertical attention maps, and fusing the weighted horizontal and vertical attention maps with the depth feature map to obtain the optimized depth feature map.

[0011] According to an example embodiment, the detection of a planar region in the input image using the depth map for region segmentation can include calculating a three-dimensional space point for planar estimation and a depth-continuous region in the input image using the depth map, and detecting a planar region in the input image using the information of the calculated three-dimensional space point and depth-continuous region for region segmentation.

[0012] According to an example embodiment, detecting the planar regions in the input image by region segmentation using the calculated three-dimensional space points and the information of the depth continuous regions can include: calculating a normal map of the input image using the calculated three-dimensional space points, and fusing the calculated normal map with a normal map estimated by the depth neural network; and segmenting the planar regions by clustering using the fused normal map and the information of the depth continuous regions.

[0013] According to an example embodiment, detecting the planar regions in the input image by region segmentation using the calculated three-dimensional space points and the information of the depth continuous regions can include: calculating a normal map of the input image using the calculated three-dimensional space points; and segmenting the planar regions by clustering using the calculated normal map and the information of the depth continuous regions.

[0014] According to an example embodiment, the depth neural network can include a feature extractor for extracting features of the input image, and a depth estimation branch for estimating depth information of the input image, wherein, in the process of estimating a depth map of the input image, a feature map of a predetermined resolution obtained by feature extraction of the input image using the feature extractor is fused with a depth feature map of the same resolution generated in the depth estimation process using the depth estimation branch, to generate the depth map using the fused depth feature map.

[0015] According to an example embodiment, the method can further include optimizing boundaries of the detected planar regions to align the boundaries of the planar regions with boundaries of real objects in the input image.

[0016] According to an example embodiment, optimizing the boundaries of the detected planar regions can include: obtaining discrete label values respectively corresponding to the detected planar regions; converting the detected planar regions into three-dimensional volumetric bodies based on the discrete label values; and optimizing the planar regions based on the converted three-dimensional volumetric bodies and the input image to align the boundaries of the planar regions with the boundaries of the real objects in the input image.

[0017] According to an example embodiment, optimizing the boundaries of the detected planar regions can include: obtaining region information respectively corresponding to each pixel point in the input image based on the detected planar regions; obtaining planar weight information of each pixel point in a four-channel image composed of a two-dimensional single-channel image composed of the region information and the input image, based on a shortest distance between each pixel point and boundaries of the detected planar regions in the two-dimensional single-channel image; determining similarity between the pixel points based on pixel values, the region information, and the planar weight information respectively corresponding to the pixel points; and performing image segmentation based on the similarity between the pixel points to obtain optimized boundaries of the planar regions.

[0018] According to another aspect of the disclosure, there is provided an apparatus for performing plane detection, the apparatus comprising: an image acquisition unit configured to acquire an input image; an estimation unit configured to extract features of the input image and estimate a depth map of the input image based on the extracted features using a deep neural network; and a region segmentation unit configured to detect a planar region in the input image using region segmentation using the depth map.

[0019] According to an example embodiment, the deep neural network can include a feature extractor for extracting features of the input image, a depth estimation branch for estimating depth information of the input image, and a normal estimation branch for estimating normal information of the input image, wherein, in estimating the depth map of the input image, the estimation unit optimizes the depth information estimated by the depth estimation branch using the normal information estimated by the normal estimation branch.

[0020] According to an example embodiment, in estimating the depth map of the input image, the estimation unit can fuse a feature map of a predetermined resolution obtained by extracting features of the input image using the feature extractor with a depth feature map of the same resolution generated in a depth estimation process using the depth estimation branch and a normal feature map of the same resolution generated in a normal estimation process using the normal estimation branch, to obtain the depth map using the fused depth feature map and the fused normal feature map.

[0021] According to an example embodiment, the optimization of the depth information estimated by the depth estimation branch using the normal information estimated by the normal estimation branch can include extracting information about a region in which a normal feature changes by more than a predetermined degree in the normal feature map and optimizing the fused depth feature map using the information to obtain an optimized depth feature map.

[0022] According to an example embodiment, the extraction of the information about the region in which the normal feature changes by more than the predetermined degree in the normal feature map and the optimization of the fused depth feature map using the information can include performing horizontal direction depth convolution and vertical direction depth convolution on the normal feature map, respectively, and obtaining a horizontal direction attention map and a vertical direction attention map for the information using an activation function, and obtaining the optimized depth feature map based on the horizontal direction attention map, the vertical direction attention map, and the depth feature map.

[0023] According to an example embodiment, the obtaining of the optimized depth feature map based on the horizontal direction attention map, the vertical direction attention map, and the depth feature map can include performing weighting processing on the horizontal direction attention map and the vertical direction attention map, and fusing the weighted horizontal direction attention map and the weighted vertical direction attention map with the depth feature map to obtain the optimized depth feature map.

[0024] According to an example embodiment, detecting the planar regions in the input image using the region segmentation with the depth map can include calculating three-dimensional space points for planar estimation and depth-continuous regions in the input image using the depth map, and detecting the planar regions in the input image using region segmentation with information of the calculated three-dimensional space points and the depth-continuous regions.

[0025] According to an example embodiment, detecting the planar regions in the input image using the region segmentation with the information of the calculated three-dimensional space points and the depth-continuous regions can include calculating a normal map of the input image with the calculated three-dimensional space points, and fusing the calculated normal map with a normal map estimated by the deep neural network, and segmenting the planar regions using clustering with the fused normal map and the information of the depth-continuous regions.

[0026] According to an example embodiment, detecting the planar regions in the input image using the region segmentation with the information of the calculated three-dimensional space points and the depth-continuous regions can include calculating a normal map of the input image with the calculated three-dimensional space points, and segmenting the planar regions using clustering with the calculated normal map and the information of the depth-continuous regions.

[0027] According to an example embodiment, the deep neural network can include a feature extractor for extracting features of the input image, and a depth estimation branch for estimating depth information of the input image, wherein the estimating unit fuses a feature map of a predetermined resolution obtained by performing feature extraction on the input image using the feature extractor with a depth feature map of the same resolution generated in a depth estimation process using the depth estimation branch, to generate the depth map using the fused depth feature map, in a process of estimating a depth map of the input image.

[0028] According to an example embodiment, the apparatus can further include a planar boundary optimization unit configured to optimize boundaries of the detected planar regions to align the boundaries of the planar regions with boundaries of real objects in the input image.

[0029] According to an example embodiment, optimizing the boundaries of the detected planar regions can include obtaining discrete label values respectively corresponding to the detected planar regions, converting the detected planar regions into three-dimensional voxels based on the discrete label values, and optimizing the planar regions based on the converted three-dimensional voxels and the input image to align the boundaries of the planar regions with the boundaries of the real objects in the input image.

[0030] According to an exemplary embodiment, the optimization of the boundary of the detected planar region can include: obtaining region information corresponding to each pixel in the input image based on the detected planar regions; obtaining planar weight information of each pixel in a four-channel image composed of the two-dimensional single-channel image composed of the region information and the input image based on the shortest distance between each pixel and the boundary of the detected planar region in the two-dimensional single-channel image; determining the similarity between the pixels based on the pixel value, the region information and the planar weight information corresponding to each pixel, respectively, and performing image segmentation based on the similarity between the pixels to obtain the optimized boundary of the planar region.

[0031] According to another aspect of the present disclosure, there is provided an electronic device, which can include a processor and a memory, wherein the memory has stored therein program instructions, wherein the program instructions, when executed by the processor, cause the processor to perform the method as described above.

[0032] According to another aspect of the present disclosure, there is provided a computer-readable recording medium having recorded thereon program instructions, wherein the program instructions, when executed by a processor, cause the processor to perform the method as described above.

[0033] According to the planar detection method and the planar detection apparatus of the present disclosure, since the planar detection is performed based on the depth map of the entire input image, the planar detection in a textureless region can be achieved. Furthermore, on this basis, the accuracy of the planar detection can be effectively improved by the feature map fusion and / or the optimization of the depth information using the normal information, and the boundary of the detected planar region can be aligned with the boundary of the real object in the input image by further performing the planar optimization operation. BRIEF DESCRIPTION OF DRAWINGS

[0034] These and / or other aspects and advantages of the present application will become apparent and more readily appreciated from the following detailed description, taken in conjunction with the accompanying drawings, in which:

[0035] Figure 1 is a schematic diagram of planar detection in an existing augmented reality framework;

[0036] Figure 2 is a schematic diagram of planar detection in an augmented reality framework according to the present disclosure;

[0037] Figure 3 is a flowchart of a method of performing planar detection according to the present disclosure;

[0038] Figure 4 is a schematic diagram of a planar detection method according to an exemplary embodiment of the present disclosure;

[0039] Figure 5is a schematic diagram illustrating operations of a deep neural network according to an example embodiment of the present disclosure;

[0040] Figure 6 is a schematic diagram illustrating operations performed by a feature fusion module according to an example embodiment of the present disclosure;

[0041] Figure 7 is a schematic diagram of a region where normal information changes significantly but depth information changes slowly;

[0042] Figure 8 is a schematic diagram of optimizing depth information using normal information according to an example embodiment of the present disclosure;

[0043] Figure 9 is a schematic diagram illustrating operations performed by a normal-guided attention module according to an example embodiment of the present disclosure;

[0044] Figure 10 is a schematic diagram illustrating an example of a planar region segmentation operation according to an example embodiment of the present disclosure;

[0045] Figure 11 is a schematic diagram illustrating an example of a planar optimization operation according to an example embodiment of the present disclosure;

[0046] Figure 12 is a schematic diagram illustrating a planar detection method according to another example embodiment of the present disclosure;

[0047] Figure 13 is a schematic diagram illustrating operations of a deep neural network according to another example embodiment of the present disclosure;

[0048] Figure 14 is a schematic diagram illustrating a planar region segmentation operation according to another example embodiment of the present disclosure;

[0049] Figure 15 is a block diagram illustrating an apparatus for performing planar detection according to the present disclosure;

[0050] Figure 16 is a schematic diagram illustrating a semantic planar detection system suitable for augmented reality according to the present disclosure. DETAILED DESCRIPTION

[0051] Before describing the inventive concept and example embodiments of the present disclosure, a brief description of a planar detection method in an existing augmented reality framework is now first provided for better understanding of the present disclosure.

[0052] Figure 1 is a schematic diagram of planar detection in an existing augmented reality framework;

[0053] As Figure 1As shown, existing plane detection methods in augmented reality frameworks heavily rely on 3D spatial point clouds from Simultaneous Localization and Mapping (SLAM) systems. Specifically, existing AR frameworks use 3D spatial point clouds from SLAM systems as input and employ the Random Sample Consensus (RANSAC) algorithm to fit plane parameters to complete plane detection. Subsequently, AR content is displayed in conjunction with camera pose. Thus, existing plane detection methods detect planes based on sparse 3D spatial point clouds generated by SLAM systems. SLAM systems first extract feature points from the input image and match the extracted 2D feature points, then use the matched 2D feature points to calculate 3D spatial points. Since 2D feature points cannot be detected in textureless regions, SLAM systems cannot obtain the corresponding 3D spatial points in textureless regions, thus preventing existing methods from performing plane detection in textureless regions. Furthermore, as mentioned above, existing plane detection methods perform plane detection based on sparse 3D spatial point clouds output by SLAM systems. However, sparse 3D point clouds cannot provide enough information to accurately estimate the boundaries of a plane, thus causing the detected plane boundaries to fail to align with the boundaries of real objects.

[0054] To address this, this disclosure proposes a novel planar detection method and apparatus. The following will refer to... Figures 2 to 15 The present disclosure provides a detailed description of the concept and exemplary embodiments for performing plane detection.

[0055] Figure 2 This is a schematic diagram of plane detection within the augmented reality framework according to this disclosure. For example... Figure 2 As shown, this disclosure directly detects planes based on the input image (which can be a color image), instead of relying on the sparse 3D point cloud output by the SLAM system for plane detection. Because it utilizes information from the entire image for plane detection, this disclosure is capable of performing plane detection in textureless regions.

[0056] To address the limitation of existing methods in detecting planes in textureless regions, this disclosure presents a deep neural network for scene information acquisition. This deep neural network provides information about the entire scene, including textureless regions, for estimating planes within the scene.

[0057] To address the issue that the detected plane cannot be aligned with the boundary of the real object, this disclosure employs deep region segmentation technology to obtain an initial plane, and uses plane boundary optimization technology to align the boundary of the detected initial plane with the boundary of the real object plane.

[0058] Below, refer to Figures 3 to 14A method of performing plane detection according to an exemplary embodiment of the present disclosure will be described in detail.

[0059] Figure 3 is a flowchart of a method of performing plane detection according to the present disclosure (hereinafter, for the convenience of description, it will be referred to as "plane detection method"). Referring to Figure 3 At step S310, an input image is acquired. For example, the input image can be acquired by a camera in response to a user request, and in addition, the input image can be acquired in real time. It should be noted that the present disclosure does not have any limitation on the way of acquiring the input image. At step S320, a feature of the input image is extracted using a deep neural network, and a depth map of the input image is estimated based on the extracted feature. The neural network can be used to acquire scene information. As an example, for example, a feature of the input image can be extracted to generate a 1 / 2 resolution feature map, a 1 / 4 resolution feature map, a 1 / 8 resolution feature map, a 1 / 16 resolution feature map, and a 1 / 32 resolution feature map of the input image, and then based on the generated feature map, various deconvolution and convolution operations of layers for information estimation in the deep neural network (for upsampling the feature map to predict a larger size of depth map and normal map) are used to finally estimate the depth map of the input image. At step S330, a plane region in the input image is detected using region segmentation using the depth map. According to an exemplary embodiment, at step S330, three-dimensional space points for plane estimation and depth continuous regions in the input image can be calculated using the depth map, and region segmentation is performed using information of the calculated three-dimensional space points and depth continuous regions to detect a plane region in the input image. Specifically, for example, three-dimensional space points can be calculated using the depth map by the following method: according to a camera imaging model (for example, a pinhole camera model), the following formula can be used to calculate three-dimensional space points:

[0060]

[0061] wherein u, v are image pixel coordinates, fx, fy are focal lengths, and cx and cy are principal points. By substituting u, v, Z into the above formula, the X, Y coordinates of the space point can be obtained, and thus all three-dimensional space points corresponding to the entire image can be obtained. Although the above method of calculating three-dimensional space points using a depth map is listed as an example, it is clear to those skilled in the art that other methods of calculating three-dimensional space points can also be used, and the specific calculation method of the three-dimensional space points is not limited by the present disclosure. In addition, for example, the depth continuous region can be calculated using a depth map by the following method: a relatively simple region growing method is used, the normal vector of each point is calculated using the three-dimensional points in the field of the point (such as 9x9), then a certain threshold (such as 10 degrees) is set, and the normal vectors of adjacent points are compared one by one, for example, if the two points are within the specified threshold, it is considered that the two points are continuous, and the average of the two points is taken as the normal vector of the merged continuous region, and then the comparison with adjacent points or other continuous regions is continued. Similarly, although the above method of calculating a depth continuous region using a depth map is listed as an example, it is clear to those skilled in the art that other methods of calculating a depth continuous region can also be used.

[0062] According to an exemplary embodiment, after the three-dimensional space points and the depth continuous region are calculated, the three-dimensional space points and the depth continuous region information can be used to perform region segmentation to detect the planar region in the input image by the following method: the normal map of the input image is calculated using the calculated three-dimensional space points, the calculated normal map is fused with the normal map estimated by the depth neural network, and the planar region is segmented by clustering using the fused normal map and the depth continuous region information. Alternatively, after the three-dimensional space points and the depth continuous region are calculated, the three-dimensional space points and the depth continuous region information can also be used to perform region segmentation to detect the planar region in the input image by the following method: the normal map of the input image is calculated using the calculated three-dimensional space points, and the planar region is segmented by clustering using the calculated normal map and the depth continuous region information. In addition, it is clear to those skilled in the art that the method of using the three-dimensional space points and the depth continuous region information to perform region segmentation to detect the planar region in the input image is not limited to the above examples.

[0063] As described above, since the planar detection method of the present disclosure estimates the depth map of the input image using the information of the entire input image, and detects the planar region in the input image using the depth map, rather than using the sparse feature points generated by the SLAM system for planar detection, it can achieve that the planar detection can be performed even in a textureless region.

[0064] Alternatively, the above planar detection method can further include the following steps: Figure 3optimizing the boundary of the detected planar region to align the boundary of the planar region with the boundary of the real object in the input image.

[0065] In the following, the content involved in the planar detection method according to the exemplary embodiments of the present disclosure will be introduced in detail. Figures 4 to 14 The content involved in the planar detection method according to the exemplary embodiments of the present disclosure will be introduced in detail.

[0066] According to an exemplary embodiment, the deep neural network mentioned in step S320 can include a feature extractor for extracting features of the input image, a depth estimation branch for estimating depth information of the input image, and a normal estimation branch for estimating normal information of the input image. However, the constitution of the deep neural network is not limited to the above example, for example, alternatively, according to another exemplary embodiment, the deep neural network can not include the normal estimation branch, but include the feature extractor for extracting features of the input image, and the depth estimation branch for estimating depth information of the input image. The present disclosure does not limit the specific constitution of the deep neural network as long as it can extract features of the input image and can estimate at least the depth map of the input image. According to the exemplary embodiment, the deep neural network can estimate not only the depth map of the input image, but also the normal map of the input image.

[0067] Figure 4 is a schematic diagram of the planar detection method according to the exemplary embodiments of the present disclosure. As shown in Figure 4 , the input image is first input into the deep neural network and then undergoes planar region segmentation and planar region optimization to output the detected planar region. In this exemplary embodiment, considering that the depth information and the surface normal information of the image have a close relationship, the deep neural network is set to include a feature extractor for extracting features of the input image, a depth estimation branch for estimating depth information of the input image, and a normal estimation branch for estimating normal information of the input image. This deep neural network can estimate the depth information and the normal information from the input single image simultaneously in a multi-task manner, thereby obtaining a dense three-dimensional point cloud for planar estimation. Specifically, the input image can be first input into the feature extractor in the deep neural network for feature extraction, and then the extracted features are input into the depth estimation branch and the normal estimation branch respectively, the depth estimation branch can perform depth estimation to output a depth map, and the normal estimation branch can perform normal estimation to output a normal map, both of which can be used in planar region segmentation. In addition, in the exemplary embodiment of Figure 4 , in the process of estimating the depth map of the input image, the normal information estimated by the normal estimation branch is used to optimize the depth information estimated by the depth estimation branch. In Figure 4In this way, the estimated depth map is more accurate, and thus the accuracy of the plane detection is improved.

[0068] In the following, the plane detection method shown in Figure 5 The depth neural network in the plane detection method shown in Figure 4 is described. Figure 5 is a schematic diagram of the operation of the depth neural network according to an example embodiment of the present disclosure. In Figure 5 In the example of the depth neural network of

[0069] Specifically, in the process of estimating the depth map of the input image, the feature maps of a predetermined resolution obtained by performing feature extraction on the input image using the feature extractor are respectively fused with the depth feature maps of the same resolution generated in the depth estimation process using the depth estimation branch and the normal feature maps of the same resolution generated in the normal estimation process using the normal estimation branch, so as to obtain the depth map using the fused depth feature maps and the fused normal feature maps. The above operation is the operation performed by the feature fusion module. Here, the predetermined resolution may, for example, be 1 / 8 and 1 / 16 resolution, but is not limited thereto. Selecting feature maps of partial resolutions (for example, feature maps of 1 / 8 and 1 / 16 resolution) for fusion can improve the accuracy and detail information of the final depth map and normal map while accelerating the training speed of the neural network. Specifically, the feature fusion module in the depth estimation branch can fuse the feature maps of a predetermined resolution obtained by performing feature extraction on the input image using the feature extractor with the depth feature maps of the same resolution generated in the depth estimation process, and the feature fusion module in the normal estimation branch can fuse the feature maps of a predetermined resolution obtained by performing feature extraction on the input image using the feature extractor with the normal feature maps of the same resolution generated in the normal estimation process (see the normal-guided attention module in the figure). Through the fusion of the feature maps, the accuracy and detail information of the final depth map and normal map can be improved, for example, the spatial details of the final results of the depth estimation and the normal estimation can be restored, so that more accurate plane detection results can be provided. In addition, the fusion of the feature maps also helps to accelerate the training speed of the depth neural network.

[0070] In the following, the plane detection method shown in Figure 6The operations performed by the feature fusion module are introduced. First, the feature map of a predetermined resolution obtained by using the input feature extractor (for example, which can be a DenseNet Encoder) and the depth feature map and the normal feature map of the same resolution can be input into the feature fusion module respectively. Second, the input feature extractor feature map is processed by using three groups (not limited to the number, three groups are only an example) of processing units composed of two-dimensional convolution, batch normalization (BN) and activation function Relu. Among them, the purpose of the first group of processing units is to reduce the channel number of the input feature map, the role of the second group of processing units is to extract the features related to the to-be-fused feature map from the feature extractor feature map and migrate them from the feature extraction feature domain to the feature domain of the to-be-fused feature, and the role of the last group of processing units is to adjust the output channel number to be equal to the channel number of the input depth feature map (or normal feature map). Then, the processed feature map output in the last step is added to the corresponding elements of the input depth feature map (or normal branch feature map). Finally, another processing unit composed of 3x3 convolution, batch normalization and activation function Relu is used to process the feature map output in the last step to obtain the fused depth feature map (or normal feature map). It should be noted that the operations performed on the feature map are not limited to being processed by the above processing units (for example, the activation function used can not be limited to Relu), and the specific way of feature map fusion is also not limited to the example shown in the above Figure 6 Any way of feature map fusion in the disclosed data can be adopted.

[0071] Next, the operations performed by the normal guided attention module in Figure 5 are introduced. As described above, in the process of estimating the depth map of the input image, the normal information estimated by the normal estimation branch is used to optimize the depth information estimated by the depth estimation branch. Specifically, for example, as shown in Figure 8 , in each stage of the depth estimation branch and the normal estimation branch, the output of the depth estimation branch can be optimized by the output of the normal estimation branch. For example, the high-frequency information (i.e., the information related to the area where the normal feature in the normal feature map changes more than a predetermined degree) in the normal feature map reflecting the normal information in the normal estimation branch can be used to optimize the depth feature map in the depth estimation branch, so that the depth feature map in the depth estimation branch remains sensitive in the area where the normal information changes obviously but the depth information changes slowly, so that the finally obtained depth map has sharp boundaries in these areas, and thus can provide more accurate three-dimensional space points for plane detection. Figure 7 is a schematic diagram of an area where the normal information changes obviously but the depth information changes slowly. As Figure 7As shown, the area marked by the circle is the area where the normal information changes obviously but the depth information changes little, and by using the high-frequency information in the normal feature map to optimize the depth feature map in the depth estimation branch, the depth feature map can remain sensitive in the area where the normal information changes obviously but the depth information changes slowly, and thus the final obtained depth map has sharp edges in these areas.

[0072] Return operation Figure 5 The optimization of the depth information estimated by the depth estimation branch using the normal information estimated by the normal estimation branch can include: extracting information related to an area in which the normal feature in the normal feature map changes by more than a predetermined degree and optimizing the depth feature map using the information to obtain an optimized depth feature map. And the extraction of the information related to the area in which the normal feature in the normal feature map changes by more than the predetermined degree and the optimization of the depth feature map using the information can include: first, respectively performing horizontal direction depth convolution and vertical direction depth convolution on the normal feature map, and obtaining a horizontal direction attention map and a vertical direction attention map for the information using an activation function; second, obtaining the optimized depth feature map based on the horizontal direction attention map, the vertical direction attention map and the depth feature map. As an example, the obtaining of the optimized depth feature map based on the horizontal direction attention map, the vertical direction attention map and the depth feature map can include: performing weighted processing on the horizontal direction attention map and the vertical direction attention map, and fusing the weighted processed horizontal direction attention map and the vertical direction attention map with the depth feature map to obtain the optimized depth feature map.

[0073] The above optimization operation is performed by the normal guided attention module shown in Figure 5 . Figure 9 is a schematic diagram showing the operation performed by the normal guided attention module according to an exemplary embodiment of the present disclosure. The above optimization operation will be described exemplarily below with reference to Figure 9 .

[0074] With reference to Figure 9 , first, the fused depth feature map and the fused normal feature map f n are input into the normal guided attention module.

[0075] Then, the horizontal direction depth convolution w h with a convolution kernel of (-1, 2, -1) T and the vertical direction depth convolution w v with a convolution kernel of (-1, 2, -1) nThe operation is performed, and a Tanh activation function is used, for example, to obtain a horizontal direction attention map and a vertical direction attention map, respectively. The horizontal direction attention map and the vertical direction attention map here are attention maps for high-frequency information in the normal feature map, and therefore, in the example shown in FIG. 6, the horizontal direction attention map and the vertical direction attention map are also respectively referred to as a "horizontal direction high-frequency attention map" and a "vertical direction high-frequency attention map". After that, the horizontal direction attention map and the vertical direction attention map are multiplied (in one way of fusion, other fusion ways can also be used) with the input depth feature map to obtain a horizontal direction attention result and a vertical direction attention result. Finally, the horizontal direction attention result, the vertical direction attention result, and the input depth branch feature map are added together with the corresponding elements using weight coefficients a and b to obtain an optimized depth feature map. In summary, the depth feature map guided by attention output by the module can be described by the following formula: Figure 9

[0076]

[0077] h v T Figure 9 Figure 9 Figure 9

[0078] Figure 5 Figure 5 ​​​​​​​​​​​​​As shown below, in the process of estimating the depth map of the input image, the normal feature map and depth feature map used to optimize the depth information estimated by the depth estimation branch using the normal information estimated by the normal estimation branch can both be fused normal feature maps and fused depth maps obtained through feature map fusion. However, in the case where the deep neural network does not include a feature map fusion module, the normal feature maps and depth feature maps used to optimize the depth information estimated by the depth estimation branch using the normal information estimated by the normal estimation branch are normal feature maps and depth feature maps that have not undergone feature map fusion.

[0079] Return to reference Figure 4 ,exist Figure 4 In an exemplary embodiment, after obtaining the depth map and normal map, the initial plane can be detected by performing planar region segmentation using the depth map and normal map. (Refer to the above...) Figure 3 As described in step S330, after estimating the depth map, the depth map can be used to calculate three-dimensional spatial points for planar estimation and depth-continuous regions in the input image. The calculated information from the three-dimensional spatial points and depth-continuous regions is then used for region segmentation to detect planar regions in the input image. And... Figure 4 When a deep neural network includes a normal estimation branch, region segmentation to detect planar regions in an input image using information from calculated 3D spatial points and depth-continuous regions can include: calculating the normal map of the input image using the calculated 3D spatial points, fusing the calculated normal map with the normal map obtained using the normal estimation branch; and then using the fused normal map and information from the depth-continuous regions to perform clustering to segment the planar regions. This operation can be referred to as "planar region segmentation." Figure 10 This is a schematic diagram illustrating an example of a planar region segmentation operation according to an exemplary embodiment of the present disclosure. Below, referring to... Figure 10 An example of planar region segmentation operation is introduced. (Refer to...) Figure 10 The planar region segmentation result can be obtained through the following steps: First, use the depth map output by the deep neural network to calculate the 3D spatial points for planar estimation and the depth-continuous regions in the input image; second, use the calculated 3D spatial points to calculate the normal map of the input image, and then fuse the calculated normal map with the normal map estimated using the deep neural network (e.g., the normal map estimated using the normal estimation branch). For example, as... Figure 10As shown, the following fusion manner can be adopted: an average normal map is obtained by using the calculated normal map and the normal map estimated by the depth estimation network, and the normal consistent region information is calculated. Finally, the fused normal map, the normal consistent region information, and the depth continuous region information are used for clustering to segment the planar region. It should be noted that the fusion of the calculated normal map and the normal map estimated by the depth neural network is not limited to the above example manner, but other fusion manners can be adopted, for example, a weighted average normal map is calculated, etc.

[0080] Referring back to Figure 4 After the planar regions are detected by the planar region segmentation, optionally, the boundaries of the planar regions can be further optimized to align the boundaries of the planar regions with the boundaries of the real objects in the input image. According to an example embodiment, the optimization of the boundaries of the detected planar regions can include the following operations: obtaining discrete label values corresponding to the detected planar regions respectively; converting the detected planar regions into three-dimensional volumetric bodies based on the discrete label values; and optimizing the planar regions based on the converted three-dimensional volumetric bodies and the input image, so as to align the boundaries of the planar regions with the boundaries of the real objects in the input image. Figure 11 is a schematic diagram showing an example of the planar optimization operation according to an example embodiment of the present disclosure. As shown, Figure 11 First, the planar regions obtained by the planar region segmentation can be numbered using the discrete label values, then the planar regions with added labels are converted into three-dimensional volumetric bodies using one-hot encoding, and finally, the converted three-dimensional volumetric bodies and the color input image are used to apply an edge-preserving optimization algorithm (such as a bilateral solver algorithm) to optimize the boundaries of the planar regions to align with the boundaries of the real objects in the scene. For example, the three-dimensional volumetric bodies can be edge-preserving optimized layer by layer to obtain the weight of each pixel on different planes after optimization, then the label with the largest weight is selected for each pixel as the optimized planar label of the pixel, and thus the planar region to which the pixel belongs can be determined.

[0081] The planar detection method according to an example embodiment of the present disclosure has been described above. However, the planar detection method of the present disclosure is not limited to the above example embodiment. Figures 4 to 11 The planar detection method according to an example embodiment of the present disclosure has been described above. However, the planar detection method of the present disclosure is not limited to the above example embodiment. Figure 12 is a schematic diagram showing a planar detection method according to another example embodiment of the present disclosure.

[0082] Referring back to Figure 12In this example embodiment, the deep neural network (may also be referred to as a depth estimation network) can include a feature extractor for extracting features of an input image, and a depth estimation branch (may also be referred to as a depth branch) for estimating depth information of the input image, but does not include a normal estimation branch (may also be referred to as a normal branch) for estimating normal information of the input image as shown in Figure 5 Since the deep neural network used in Figure 12 contains only the depth estimation branch and the feature extractor but does not include the normal estimation branch, the amount of calculation is smaller, the depth information can be obtained in real time, and therefore it is advantageous to deploy on a mobile device.

[0083] Figure 13 is a schematic diagram showing the operation of the deep neural network according to this example embodiment. As shown in Figure 13 , the feature fusion module mentioned above is included in the depth estimation branch. Specifically, in the process of estimating the depth map of the input image, the feature map of a predetermined resolution (for example, a feature map of 1 / 16 resolution) obtained by performing feature extraction on the input image using the feature extractor is fused with the depth feature map of the same resolution generated in the depth estimation process using the depth estimation branch, to generate the depth map using the fused depth feature map. Through the fusion of the feature maps, the accuracy and the detail information of the final depth map can be improved, so that a more accurate plane detection result can be provided.

[0084] For example, the feature map generated by the feature extractor and the depth feature map can be fused in the manner shown in (a) of Figure 7 to obtain the fused depth feature map, and the final depth map can be generated using the fused depth map.

[0085] After that, the plane can be detected by plane region segmentation using the depth map. Figure 14 is a schematic diagram showing the operation of plane region segmentation according to another example embodiment of the present disclosure. Referring to Figure 14 , the three-dimensional space points for plane estimation and the depth continuous regions in the input image can be first calculated using the depth map, and then the plane regions in the input image can be detected by region segmentation using the information of the calculated three-dimensional space points and the depth continuous regions. For example, as shown in Figure 14 , detecting the plane regions in the input image by region segmentation using the information of the calculated three-dimensional space points and the depth continuous regions can include calculating the normal map of the input image using the calculated three-dimensional space points, and segmenting the plane regions by clustering using the calculated normal map and the information of the depth continuous regions.

[0086] Referring back to Figure 13Optionally, after detecting the planar regions by the planar region segmentation, the planar detection method according to another exemplary embodiment of the present disclosure can continue to optimize the boundaries of the detected planar regions so as to align the boundaries of the planar regions with the boundaries of the real objects in the input image. For example, the planar optimization technique based on image segmentation can be adopted, but is not limited thereto, for example, the planar optimization technique based on image segmentation described above with reference to Figure 11 the planar detection method shown in FIG. 6 can also be applied to the planar optimization. Likewise, the planar optimization technique based on image segmentation described below can also be applied to the planar optimization of the planar detection method shown in FIG. 6. Figure 4 As an example, the optimization of the boundaries of the detected planar regions can include: obtaining region information corresponding to each pixel in the input image based on the detected planar regions; obtaining planar weight information of each pixel in a four-channel image composed of the two-dimensional single-channel image composed of the region information and the input image based on the shortest distances between each pixel and the boundaries of the detected planar regions in the two-dimensional single-channel image; determining the similarity between the pixels based on the pixel values, the region information and the planar weight information corresponding to each pixel, respectively, and performing image segmentation based on the similarity between the pixels to obtain the optimized planar region boundaries.

[0087] Specifically, a digital label can be added to each planar region as a number, and a two-dimensional single-channel image composed of the planar region numbers can be taken as the fourth channel of the color image. The four-channel image is segmented, and the segmentation result is used to optimize the planar region boundaries. Through the above planar optimization operation, the detected planar regions can be aligned with the boundaries of the real objects.

[0088] The planar optimization based on image segmentation will be described below.

[0089] First, the planar segmentation module can be used to number each planar region with a discrete value label, and a two-dimensional single-channel image composed of the planar region numbers can be taken together with the scene color image to form a four-channel image. At this time, the value of each pixel of the four channels is [R, G, B, P], where R is the red channel, G is the green channel, B is the blue channel, and P is the channel composed of the planar region label numbers, where the P value can reflect the region information corresponding to each pixel. Then, the planar weight map W plane is calculated using the two-dimensional single-channel image composed of the planar region numbers. The weight value at each pixel in the weight map is proportional to the shortest distance between the pixel and the boundary of the planar region in the two-dimensional single-channel image composed of the planar region numbers. Thus, the planar weight information of each pixel in the four-channel image can be obtained based on the shortest distances between each pixel and the boundaries of the detected planar regions in the two-dimensional single-channel image composed of the planar region numbers.

[0090] Finally, the image segmentation algorithm (e.g., Efficient Graph-based image segmentation algorithm) is used on the four-channel image generated in the first step to perform image segmentation based on a pixel similarity function, and the optimized planar region boundary is obtained. The pixel similarity function is defined as follows:

[0091]

[0092] wherein pixel1 and pixel2 are two pixels to be calculated for similarity, r1, g1, b1, and p1 are pixel values of pixel1 in the four-channel image, r2, g2, b2, and p2 are pixel values of pixel2 in the four-channel image, w1 and w2 are weight values of pixel1 and pixel2 in the weight map W plane

[0093] The above describes some exemplary embodiments of the planar detection method of the present disclosure. The planar detection method of the present disclosure can detect a planar in a non-texture region because it is based on the depth map of the entire input image. In addition, the feature map fusion module and / or the normal guided attention module described above can effectively improve the accuracy of planar detection, and by further performing a planar optimization operation, the boundary of the detected planar region can be aligned with the boundary of the real object in the input image. In AR applications, it is often necessary to place virtual objects in a real scene, and there are many non-texture regions (such as a wall with a pure color, a table top, etc.) in the real scene. By using the planar detection method according to the present disclosure, a planar can be detected in a non-texture region, so that the user can place a virtual object in such a non-texture region, meeting the user's needs and improving the user experience. In addition, for example, in AR games, it is often necessary for virtual objects to interact with real objects. By using the planar detection method according to the present disclosure, a planar detection result that is aligned with the boundary of a real object can be provided, so that the accuracy of the interaction between a virtual object and a real object can be improved, and the gaming experience can be improved.

[0094] The planar detection method of the present disclosure can be applied in AR glasses, smart phones or other AR terminals, and can also be applied in navigation, exhibition, training, gaming and other applications.

[0095] Figure 15 is a block diagram showing an apparatus for performing planar detection according to the present disclosure (hereinafter, for the sake of convenience, referred to as "planar detection apparatus").

[0096] Referring to Figure 15 ​The planar surface detection apparatus 1500 can include an image acquisition unit 1510, an estimation unit 1520, and a region segmentation unit 1530. Specifically, the image acquisition unit 1510 can be configured to acquire an input image. The estimation unit 1520 can be configured to extract features of the input image using a deep neural network and estimate a depth map of the input image based on the extracted features. The region segmentation unit 1530 can be configured to detect a planar region in the input image using region segmentation with the depth map. Specifically, for example, the region segmentation unit 1530 can calculate three-dimensional space points for planar surface estimation and depth-continuous regions in the input image using the depth map and detect a planar region in the input image using region segmentation with information of the calculated three-dimensional space points and depth-continuous regions. Optionally, the planar surface detection apparatus 1500 can further include a planar surface boundary optimization unit (not shown) that can optimize boundaries of the detected planar region to align the boundaries of the planar region with boundaries of a real object in the input image. Since the above-mentioned operations involved or details thereof have been described in the above in the description of the process of the planar surface detection method according to the present disclosure, they will not be described herein again for the sake of brevity, and the corresponding contents or details can be referred to the description of the planar surface detection method according to the present disclosure. Figures 3 to 14

[0097] The planar surface detection method and the planar surface detection apparatus according to the embodiments of the present disclosure have been described above with reference to Figures 1 to 15 However, it should be understood that: Figure 15 Each unit in the apparatus shown can be configured as software, hardware, firmware, or any combination thereof, which performs a specific function. For example, these units can correspond to a dedicated integrated circuit, a pure software code, or a module combining software and hardware. As an example, the apparatus described with reference to Figure 15 may be a PC computer, a tablet apparatus, a personal digital assistant, a smart phone, a web application, or other apparatus capable of executing program instructions, but is not limited thereto.

[0098] It should be noted that although the planar surface detection apparatus 1400 is divided into units for performing respective processes in the above, it is clear to those skilled in the art that the processes performed by each unit can also be performed without any specific unit division or clear demarcation between units. In addition, the apparatus described above with reference to Figure 15 is not limited to including the units described above, but can also add some other units (e.g., a storage unit, a data processing unit, etc.) as needed, or the above units can be combined.

[0099] As Figure 16 ​As shown, the semantic plane detection system suitable for augmented reality proposed by the present disclosure contains three parts, the first is a deep neural network for scene information acquisition, the second is a plane region segmentation module, and the third is a plane boundary optimization module.

[0100] In the first implementation scheme, the deep neural network is used to estimate the depth and normal information, and a normal guided attention module is designed in the network structure of this module, which utilizes the high frequency information in the normal feature map to make the estimated depth map more sharp at the boundary, so as to provide more accurate dense three-dimensional space point cloud for plane detection; the dense three-dimensional space point cloud containing normal information is clustered and segmented in the plane region segmentation module, and more accurate and robust plane regions are obtained; in the plane boundary optimization module, a boundary preserving optimization algorithm is used to optimize the obtained plane regions to align them with the real object boundaries, and semantic planes are obtained.

[0101] In the second implementation scheme, first, the input image is input into the depth estimation network to obtain the depth information of the whole image, then the obtained depth map is input into the plane region segmentation module to obtain the plane region segmentation result, and finally, the plane optimization method based on image segmentation is used to make the boundary of the plane region better align with the real object boundary, wherein the depth estimation network can provide sufficient information for the subsequent plane region segmentation module, so that it can calculate the plane region on the whole image including the textureless region, the plane region segmentation module and the subsequent plane optimization module based on image segmentation work together to make the boundary of the detected plane region better align with the real object boundary of the scene, and the plane optimization algorithm is efficient.

[0102] In addition, the plane detection method according to the present disclosure can be recorded in a computer readable recording medium. Specifically, according to the present disclosure, a computer readable recording medium having program instructions recorded thereon can be provided, which, when executed by a processor, can cause the processor to perform the plane detection method as described above. Examples of computer readable recording media can include magnetic media (such as hard disks, floppy disks, and magnetic tapes); optical media (such as CD-ROM and DVD); magneto-optical media (such as optical disks); and hardware devices specially configured to store and execute program instructions (such as read-only memory (ROM), random access memory (RAM), flash memory, etc.). In addition, according to the present disclosure, an electronic device including a processor and a memory having program instructions stored therein can also be provided, wherein when the program instructions are executed by the processor, the processor is caused to perform the plane detection method as described above. Examples of program instructions include, for example, machine code generated by a compiler and files containing high-level code that can be executed by a computer using an interpreter.

[0103] In addition, some operations in the plane detection method according to the example embodiments of the present application can be implemented by software, and some operations can be implemented by hardware, and in addition, these operations can also be implemented by a combination of software and hardware.

[0104] In addition, the present disclosure also provides an electronic device, which comprises a processor and a memory, wherein the memory stores program instructions, and when the program instructions are executed by the processor, the processor executes the plane detection method according to the present disclosure. As an example, the electronic device can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above-mentioned instruction set. Here, the electronic device does not necessarily have to be a single electronic device, but can also be any collection of devices or circuits that can execute the above-mentioned instructions (or instruction sets) individually or jointly. The electronic device can also be part of an integrated control system or system manager, or can be configured as a portable electronic device that interfaces with local or remote (e.g., via wireless transmission).

[0105] Although the present disclosure has been particularly shown and described with reference to the example embodiments thereof, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims.

Claims

1. A method for performing plane detection, comprising: Obtain the input image; Using deep neural networks, features of the input image are extracted and the depth map of the input image is estimated based on the extracted features; The depth map is used for region segmentation to detect planar regions in the input image. The use of the depth map for region segmentation to detect planar regions in the input image includes: The depth map is used to calculate 3D spatial points for planar estimation and depth-continuous regions in the input image, and the calculated 3D spatial points and depth-continuous regions are used for region segmentation to detect planar regions in the input image.

2. The method as described in claim 1, wherein, The deep neural network includes a feature extractor for extracting features from the input image, a depth estimation branch for estimating the depth information of the input image, and a normal estimation branch for estimating the normal information of the input image. In the process of estimating the depth map of the input image, the depth information estimated by the depth estimation branch is optimized using the normal information estimated by the normal estimation branch.

3. The method as described in claim 2, wherein, In the process of estimating the depth map of the input image, the feature map of a predetermined resolution obtained by extracting features from the input image using the feature extractor is fused with the depth feature map of the same resolution generated during the depth estimation process using the depth estimation branch and the normal feature map of the same resolution generated during the normal estimation process using the normal estimation branch, so as to obtain the depth map using the fused depth feature map and the fused normal feature map.

4. The method of claim 3, wherein, Optimizing the depth information estimated by the depth estimation branch using the normal information estimated by the normal estimation branch includes: Information related to regions in the normal feature map where the change in normal features exceeds a predetermined degree is extracted, and the information is used to optimize the depth feature map to obtain an optimized depth feature map.

5. The method of claim 4, wherein, Extracting information related to regions in the normal feature map where the change in normal features exceeds a predetermined degree, and using that information to optimize the depth feature map, includes: The normal feature map is subjected to horizontal and vertical depth convolutions respectively, and the horizontal and vertical attention maps for the information are obtained by using activation functions. An optimized depth feature map is obtained based on the horizontal attention map, the vertical attention map, and the depth feature map.

6. The method of claim 5, wherein, The optimized depth feature map obtained based on the horizontal attention map, vertical attention map, and depth feature map includes: The horizontal and vertical attention maps are weighted. The weighted horizontal and vertical attention maps are fused with the depth feature map to obtain the optimized depth feature map.

7. The method of claim 1, wherein, Detecting planar regions in an input image by using calculated information from three-dimensional spatial points and depth-continuous regions for region segmentation includes: The normal map of the input image is calculated using the calculated 3D spatial points, and then fused with the normal map estimated by the deep neural network. The planar region is segmented by clustering using the fused normal map and information from the depth continuous region.

8. The method of claim 1, wherein, Detecting planar regions in an input image by using calculated information from three-dimensional spatial points and depth-continuous regions for region segmentation includes: Calculate the normal map of the input image using the calculated 3D spatial points; The planar regions are segmented by clustering using the calculated normal map and information from the depth continuous region.

9. The method of claim 8, wherein, The deep neural network includes a feature extractor for extracting features from the input image, and a depth estimation branch for estimating the depth information of the input image. In the process of estimating the depth map of the input image, a feature map of a predetermined resolution obtained by using a feature extractor to extract features from the input image is fused with a depth feature map of the same resolution generated during the depth estimation process using a depth estimation branch, so as to generate the depth map using the fused depth feature map.

10. The method of any one of claims 1 to 9, further comprising: The boundaries of the detected planar regions are optimized to align with the boundaries of real objects in the input image.

11. The method of claim 10, wherein optimizing the boundary of the detected planar region comprises: Obtain the discrete label values ​​corresponding to each detected planar region; Based on discrete label values, the detected planar regions are converted into three-dimensional volumes; Based on the transformed 3D volume and the input image, the planar region is optimized so that the boundary of the planar region is aligned with the boundary of the real object in the input image.

12. The method of claim 10, wherein optimizing the boundary of the detected planar region comprises: Based on the detected planar regions, obtain the region information corresponding to each pixel in the input image; Based on the shortest distance between each pixel and the boundary of each detected planar region in the two-dimensional single-channel image composed of region information, the planar weight information of each pixel in the four-channel image composed of the two-dimensional single-channel image and the input image is obtained. Based on the pixel value, region information, and planar weight information corresponding to each pixel, the similarity between pixels is determined, and image segmentation is performed based on the similarity between pixels to obtain the optimized planar region boundary.

13. An apparatus for performing planar detection, comprising: The image acquisition unit is configured to acquire the input image; The estimation unit is configured to use a deep neural network to extract features from the input image and estimate the depth map of the input image based on the extracted features; The region segmentation unit uses the depth map to perform region segmentation to detect planar regions in the input image. The region segmentation unit is configured to use the depth map to calculate three-dimensional spatial points for planar estimation and depth-continuous regions in the input image, and to use the information of the calculated three-dimensional spatial points and depth-continuous regions to perform region segmentation to detect planar regions in the input image.

14. An electronic device comprising a processor and a memory, wherein, The memory stores program instructions, wherein when the program instructions are executed by the processor, they cause the processor to perform the method as described in any one of claims 1 to 12.

15. A computer-readable recording medium, wherein, The computer-readable recording medium contains program instructions, wherein when the program instructions are executed by a processor, they cause the processor to perform the method as described in any one of claims 1 to 12.

Images