Planar detection method and apparatus therefor

Abstract

The application discloses a plane detection method and device, and belongs to the technical field of image processing. The plane detection method comprises the following steps: acquiring a first depth image collected by a camera of an electronic device at a first time in a shooting scene; performing down-sampling on the first depth image to obtain a down-sampled image; detecting a first plane of the shooting scene at the first time according to the down-sampled image; performing up-sampling on the first plane, and adjusting a region boundary of the first plane according to the up-sampling result; and detecting a plane of the shooting scene at a second time according to the first plane, a second depth image collected by the camera at the second time in the shooting scene, and a pose change of the camera from the first time to the second time, wherein the second time is after the first time.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a plane detection method and a plane detection device. BACKGROUND

[0002] The plane detection technology is crucial for realizing real scene three-dimensional reconstruction, augmented reality, indoor navigation and other applications. When the plane detection technology is applied to a terminal device (for example, a mobile phone), the use experience of the terminal device can be greatly improved.

[0003] At present, when the plane detection is performed, a Hough-based plane detection algorithm can be used, that is, the original space plane is converted into a parameter space peak value problem. However, this method needs to convert all points into a parameter space and accumulate sorting, and therefore, the time and space complexity is relatively high. The terminal device cannot support such a complex plane detection algorithm. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a plane detection method and a plane detection device, which can solve the problem that the terminal cannot detect a plane with high complexity in the prior art.

[0005] In a first aspect, the embodiments of the present application provide a plane detection method, and the method comprises the following steps:

[0006] obtaining a first depth image collected by a camera of an electronic device at a first time in a shooting scene;

[0007] performing down-sampling on the first depth image to obtain a down-sampled image;

[0008] detecting a first plane of the shooting scene at the first time according to the down-sampled image;

[0009] performing up-sampling on the first plane, and adjusting a region boundary of the first plane according to a result of the up-sampling;

[0010] detecting a plane of the shooting scene at a second time according to the first plane, a second depth image collected by the camera at the second time in the shooting scene, and a pose change of the camera from the first time to the second time;

[0011] The second time is after the first time.

[0012] In a second aspect, the embodiments of the present application provide a plane detection device, and the device comprises:

[0013] an image obtaining module, configured to obtain a first depth image collected by a camera of an electronic device at a first time in a shooting scene;

[0014] down-sampling module, configured to down-sample the first depth image to obtain a down-sampled image;

[0015] a first plane detecting module, configured to detect a first plane of the shooting scene at the first time according to the down-sampled image;

[0016] a boundary adjusting module, configured to up-sample the first plane, and adjust a region boundary of the first plane according to a result of the up-sampling;

[0017] a second plane detecting module, configured to detect a plane of the shooting scene at a second time according to the first plane, a second depth image collected by the camera at the second time in the shooting scene, and a pose change of the camera from the first time to the second time;

[0018] wherein the second time is after the first time.

[0019] In a third aspect, an electronic device is provided, which includes a processor and a memory. The memory stores programs or instructions executable on the processor. When the programs or instructions are executed by the processor, the steps of the method according to the first aspect are implemented.

[0020] In a fourth aspect, a readable storage medium is provided, which stores programs or instructions. When the programs or instructions are executed by a processor, the steps of the method according to the first aspect are implemented.

[0021] In a fifth aspect, a chip is provided, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is configured to run programs or instructions to implement the method according to the first aspect.

[0022] In a sixth aspect, a computer program product is provided, which is stored in a storage medium. The computer program product is executed by at least one processor to implement the method according to the first aspect.

[0023] In the embodiments of the present application, the first depth image collected by the camera of the electronic device in the shooting scene at the first time can be obtained, and the first depth image is down-sampled to obtain a down-sampled image. Then, the first plane of the shooting scene at the first time is detected according to the down-sampled image. Furthermore, the first plane is up-sampled, and the region boundary of the first plane is adjusted according to the result of the up-sampling. Furthermore, the plane of the shooting scene at the second time is detected according to the first plane, the second depth image collected by the camera in the shooting scene at the second time, and the pose change of the camera from the first time to the second time.

[0024] The downsampling can reduce the size of the first depth image, so that the processing data in the plane detection process can be reduced, and the processing complexity of the plane detection can be reduced. On this basis, the detected first plane is further upsampled, and the region boundary of the first plane is adjusted according to the upsampled result, so that the accuracy of the plane detection can be met on the basis of reducing the processing complexity. And on the basis of the second depth image and the detected first plane, the plane of the shooting scene at the second time is detected in combination with the pose change of the camera, so that the probability of repeated plane detection can be reduced, the complexity of the plane detection can be reduced, and the detection of the complex plane can be realized. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a flowchart of a plane detection method of an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of downsampling in an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of a convex hull of an exemplary point set in an embodiment of the present application;

[0028] Figure 4 is a schematic diagram of plane updating in an embodiment of the present application;

[0029] Figure 5 is a flowchart of a specific implementation of a plane detection method of an embodiment of the present application;

[0030] Figure 6 is a structural block diagram of a plane detection device of an embodiment of the present application;

[0031] Figure 7 is one of the schematic block diagrams of an electronic device of an embodiment of the present application;

[0032] Figure 8 is the second schematic block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be described clearly in combination with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0034] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of a kind and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in an "or" relationship.

[0035] The planar detection method provided by the embodiments of the present application will be described in detail below in combination with the drawings, specific embodiments and application scenarios.

[0036] In a first aspect, the embodiments of the present application provide a planar detection method, as shown in the method can include the following steps: Figure 1

[0037] Step 101: Obtain a first depth image collected by a camera of an electronic device at a first time in a shooting scene.

[0038] Wherein, the depth image is also called distance image, which refers to the distance from the image collector to each point in the scene as the pixel value of the image.

[0039] In the embodiments of the present application, the camera can collect the depth image of the shooting scene. Optionally, the camera can be a camera based on time of flight (TOF).

[0040] Wherein, TOF technology is a high-precision, high-frame-rate technology in optoelectronic ranging, that is, the laser pulses generated by the laser are used for optoelectronic ranging, and the distances between all objects in the scene and the TOF-based camera are calculated. On this basis, by integrating these distance information, a relatively accurate depth image can be obtained.

[0041] Step 102: Downsample the first depth image to obtain a downsampled image.

[0042] Wherein, downsampling is to reduce the number of sampling points. For an a*b image, if the downsampling coefficient is K i , then every K i point in each row and each column of the original image is taken to form an image. Optionally, the depth image can be downsampled by using a Gaussian pyramid.

[0043] For example Figure 2 ​As shown, the original depth map 201 of size 640*480 is firstly blurred by Gaussian filtering, and then the blurred image is down-sampled by two layers with fixed ratios, K1=2, i.e. taking one point every two points in both horizontal and vertical directions; K2=4, i.e. taking one point every four points in both horizontal and vertical directions. After down-sampling, a down-sampled image 202 of size 320*240 and a down-sampled image 203 of size 160*120 are obtained.

[0044] Step 103: detecting a first plane of the shooting scene at the first time according to the down-sampled image.

[0045] In step 103, the first plane includes at least one of a horizontal plane and a vertical plane.

[0046] In the above method, the down-sampling can reduce the size of the depth image, so that the processing data in the plane detection process can be reduced, and the processing complexity of the plane detection can be reduced.

[0047] Step 104: up-sampling the first plane, and adjusting the region boundary of the first plane according to the up-sampling result.

[0048] In the above method, the up-sampling is to restore an image from a lower size to a larger size. Therefore, the up-sampling can improve the resolution of the image. As can be seen, on the basis of steps 101 to 103, the region boundary of the first plane is adjusted according to the up-sampling result by further up-sampling the first plane, so that the accuracy of the plane detection can be met on the basis of reducing the processing complexity.

[0049] Optionally, the up-sampling can be implemented in an interpolation manner.

[0050] Step 105: detecting a plane of the shooting scene at a second time according to the first plane, a second depth image of the shooting scene collected by the camera at the second time, and a pose change of the camera from the first time to the second time.

[0051] In the above method, the second time is after the first time.

[0052] In addition, when the same camera collects images of the shooting scene at different times, the angle of the collected image can change due to the pose change of the camera, but the same plane exists in the collected image. Therefore, the first plane detected at the first time also exists in the second depth image of the shooting scene collected at the second time. Based on this, the plane of the shooting scene at the second time is detected based on the above second depth image and the detected first plane, combined with the pose change of the camera, so that the probability of repeated plane detection can be reduced, and the complexity of the plane detection and the detection time can be reduced.

[0053] In addition, after detecting the plane of the shooting scene, a three-dimensional reconstruction of the shooting scene can be performed based on the detected plane to complete tasks such as modeling and target tracking.

[0054] As can be seen from the above steps 101 to 105, in the embodiment of the present application, the first depth image of the shooting scene collected by the camera of the electronic device at the first time can be obtained, so as to down-sample the first depth image to obtain a down-sampled image, and detect the first plane of the shooting scene at the first time according to the down-sampled image. Then, the first plane is up-sampled, and the region boundary of the first plane is adjusted according to the up-sampling result. Then, the plane of the shooting scene at the second time is detected according to the first plane, the second depth image of the shooting scene collected by the camera at the second time, and the pose change of the camera from the first time to the second time.

[0055] The down-sampling can reduce the size of the first depth image, so as to reduce the processing data in the plane detection process, and further reduce the processing complexity of the plane detection. On this basis, the region boundary of the first plane is adjusted according to the up-sampling result of the detected first plane, so as to meet the accuracy of the plane detection on the basis of reducing the processing complexity. Moreover, the plane of the shooting scene at the second time is detected according to the second depth image and the detected first plane, and in combination with the pose change of the camera, which can reduce the probability of repeated detection of the plane, so as to reduce the complexity of the plane detection. Therefore, the plane detection method of the embodiment of the present application not only can meet the detection accuracy, but also has a relatively low complexity, and can be applied to a terminal device.

[0056] Optionally, the step 103 “detecting the first plane of the shooting scene at the first time according to the down-sampled image” includes the following steps A-1 to A-4:

[0057] Step A-1: calculating the first coordinates of each pixel point in the down-sampled image in the camera coordinate system;

[0058] Step A-2: clustering each pixel point in the down-sampled image according to the first coordinates to obtain at least one cluster;

[0059] Step A-3: performing convex hull detection on each cluster to obtain a convex hull corresponding to each cluster;

[0060] Step A-4: performing plane fitting on the pixel points on the convex hull to obtain a fitting plane corresponding to each convex hull, and determining the fitting plane as the first plane.

[0061] Therefore, when detecting the first plane of the shooting scene at the first time according to the down-sampling image, the first coordinates of each pixel point in the camera coordinate system can be calculated first, so that each pixel point is clustered according to the first coordinates, and then the convex hull detection is performed on each clustering cluster obtained, to obtain a plurality of convex hulls, and then the plane fitting is performed on the pixel points on the convex hulls, to obtain a plurality of first planes. Each clustering cluster obtained in the process corresponds to a convex hull, and each convex hull corresponds to a first plane fitted.

[0062] The specific implementation process of each step in steps A-1 to A-4 will be described below as described in 1.1 to 1.4:

[0063] 1.1 For step A-1:

[0064] Optionally, the above step A-1 “calculating the first coordinates of each pixel point in the down-sampling image in the camera coordinate system” comprises:

[0065] According to the depth information in the down-sampling image and the parameter information (such as the intrinsic coefficient) of the camera, the first coordinates of each pixel point in each down-sampling image in the camera coordinate system are calculated.

[0066] Further, the first coordinates can be calculated according to the (1) formula;

[0067]

[0068] Wherein, (X i , Y i , Z i ) represents the first coordinates of the i-th pixel point in the down-sampling image in the camera coordinate system, (u i v i ) represents the coordinates of the i-th pixel point in the down-sampling image in the pixel coordinate system, depth i represents the depth information of the i-th pixel point in the down-sampling image, which can be the gray value in the depth map (optionally, depth i can be multiplied by 1×10 -3 , so as to convert the unit to meters (m));

[0069] (f x , f y ) represents the intrinsic coefficient of the camera-focal length, (u0, v0) represents the intrinsic coefficient of the camera-principal point coordinates;

[0070] K i represents the scale factor of the down-sampling.

[0071] It should be noted that the first coordinates can also be converted to a world coordinate system by a rotation matrix R and a translation vector T provided by simultaneous localization and mapping (SLAM) of a camera, so as to facilitate other processing in the world coordinate system, such as three-dimensional reconstruction, modeling and target tracking of a scene.

[0072] 1.2 For step A-2:

[0073] Optionally, the step A-2 “cluster each pixel point in the down-sampling image according to the first coordinates to obtain at least one cluster” includes the following steps B-1 to B-3:

[0074] Step B-1: divide each pixel point in the down-sampling image into at least one of a horizontal plane pixel point and a vertical plane pixel point according to the first coordinates;

[0075] Step B-2: establish a nearest neighbor K-dimensional tree for at least one of the horizontal plane pixel point and the vertical plane pixel point;

[0076] Step B-3: cluster the pixel points in the nearest neighbor K-dimensional tree to obtain at least one cluster.

[0077] Among them, the horizontal plane pixel point is a pixel point belonging to the horizontal plane; the vertical plane pixel point is a pixel point belonging to the vertical plane.

[0078] It should be noted that in step B-1, the pixel points are divided into horizontal plane pixel points, then in step B-2, a nearest neighbor K-dimensional tree is established for the horizontal plane pixel points, and in step B-3, the nearest neighbor K-dimensional tree established for the horizontal plane pixel points is clustered; in step B-1, the pixel points are divided into vertical plane pixel points, then in step B-2, a nearest neighbor K-dimensional tree is established for the vertical plane pixel points, and in step B-3, the nearest neighbor K-dimensional tree established for the vertical plane pixel points is clustered; in step B-1, the pixel points are divided into horizontal plane pixel points and vertical plane pixel points, then in step B-2, a nearest neighbor K-dimensional tree is established for the horizontal plane pixel points and the vertical plane pixel points respectively, and in step B-3, the nearest neighbor K-dimensional tree established for the horizontal plane pixel points and the nearest neighbor K-dimensional tree established for the vertical plane pixel points are clustered respectively.

[0079] That is, a nearest neighbor K-dimensional tree is established for the horizontal plane pixel points, and a nearest neighbor K-dimensional tree is established for the vertical plane pixel points.

[0080] As can be known from steps B-1 to B-3, in the embodiment of the present application, when clustering each pixel point in the down-sampled image, each pixel point can be first divided into horizontal plane pixel points and vertical plane pixel points, and then the horizontal plane pixel points and the vertical plane pixel points are processed respectively. In this way, in the embodiment of the present application, the horizontal plane and the vertical plane can be detected, so that the plane detection method of the embodiment of the present application can be applicable to the case of a ladder structure or a relatively complex plane.

[0081] The specific implementation process of each step in steps B-1 to B-3 is described below as 1.2.1 to 1.2.3.

[0082] 1.2.1: Optionally, the step B-1 “according to the first coordinates, dividing each pixel point in the down-sampled image into at least one of horizontal plane pixel points and vertical plane pixel points” comprises steps C-1 to C-4 as follows:

[0083] Step C-1: According to the first coordinates, calculating a target normal vector of each pixel point in the down-sampled image in the camera coordinate system;

[0084] Step C-2: Calculating an included angle between the target normal vector and a horizontal plane normal vector in the camera coordinate system;

[0085] Step C-3: In the case that the included angle corresponding to the pixel point is less than a first threshold value, the pixel point is divided into the horizontal plane pixel point;

[0086] Step C-4: In the case that the included angle corresponding to the pixel point is greater than a second threshold value, the pixel point is divided into the vertical plane pixel point.

[0087] In step C-1, a weight-based neighborhood method can be used to obtain the normal vector of a spatial point, and the specific process can be as follows: the normal vector of each pixel point is calculated according to the (2) formula;

[0088]

[0089] Here, n i represents the normal vector of the i-th pixel point in the down-sampled image in the camera coordinate system, k is the number of points selected by the nearest neighbor algorithm, for example, can be 4, 8, 12, w j is the weight of the region normal vector composed of points p i，j , p i and q i，j+1 ; p i represents the i-th pixel point in the down-sampled image, q i，j represents the j-th pixel point around p i in the counterclockwise direction, qi，j+1 p i The (j+1)th pixel in the counter-clockwise direction.

[0090] Among them, w j It can also be calculated by angle, for example, according to formula (3);

[0091]

[0092] In addition, in step C-2, the target normal vector n of the i-th pixel can be calculated according to formula (4). i The angle θ between the camera and the horizontal plane normal vector n0 in the camera coordinate system i .

[0093]

[0094] From steps C-3 to C-4 above, it can be seen that if θ i <θ Tmin θ Tmin If θ is the first threshold, then the i-th pixel belongs to the horizontal plane pixel; if θ i >θ Tmax θ Tmax If θ is the second threshold, then the i-th pixel belongs to the vertical plane pixel; if θ Tmin <θ i <θ Tmax If the i-th pixel does not belong to either the horizontal plane or the vertical plane, then the i-th pixel does not belong to either the horizontal plane or the vertical plane.

[0095] 1.2.2: Optionally, step B-2 above, "constructing a nearest neighbor K-dimensional tree for at least one of the pixels in the horizontal plane and the pixels in the vertical plane," includes the following steps D-1 to D-4:

[0096] Step D-1: Calculate the variance of the coordinate values ​​of the first pixel in the three dimensions of the camera coordinate system, and obtain the maximum variance. The first pixel includes at least one of the pixels in the horizontal plane and the pixels in the vertical plane.

[0097] Step D-2: Select the median of the target dimension to which the maximum variance belongs as the sample split point;

[0098] Step D-3: In the first pixel, the pixels whose coordinate values ​​in the target dimension are less than the sample partitioning point are divided into the left subtree, and the pixels whose coordinate values ​​in the target dimension are greater than or equal to the sample partitioning point are divided into the right subtree.

[0099] Step D-4: Construct a nearest neighbor K-dimensional tree for the pixels of the left subtree and the pixels of the right subtree respectively.

[0100] In the K-dimensional tree structure, a K-dimensional tree structure is a tree-shaped data structure for storing instance points in a K-dimensional space for fast retrieval, and is mainly applied to search of key data in a multi-dimensional space. The construction process of the nearest neighbor K-dimensional tree includes four steps of selecting a dimension (step D-1), selecting a median (step D-2), splitting data (step D-3), and recursive iteration (step D-4).

[0101] In the embodiment of the present application, when the dimension is selected, all the first pixel points have corresponding coordinate values in the three dimensions of the X axis, the Y axis, and the Z axis. Therefore, a variance can be calculated for the coordinate values of all the first pixel points in the X axis, a variance can be calculated for the coordinate values of all the first pixel points in the Y axis, and a variance can be calculated for the coordinate values of all the first pixel points in the Z axis. Thus, a maximum variance can be selected from the three variances, and the dimension to which the maximum variance belongs is the selected dimension for constructing the nearest neighbor K-dimensional tree.

[0102] Secondly, for example, if the dimension to which the maximum variance belongs is the X axis, the median of the coordinate values of all the first pixel points in the X axis (i.e., the coordinate value located in the middle position after being arranged in ascending order) is taken as the sample division point. When the dimension to which the maximum variance belongs is the Y axis or the Z axis, the sample division point is selected in the same way.

[0103] Thirdly, all the first pixel points can be divided into a left sub-tree or a right sub-tree according to the selected sample division point. Then, the process of selecting a dimension, selecting a median, splitting data, and recursive iteration is repeated for the pixel points in the left sub-tree and the right sub-tree until the obtained sub-tree includes only one pixel point.

[0104] 1.2.3: In step B-3, the agglomerative hierarchical clustering method can be used for clustering processing, that is, for one of the nearest neighbor K-dimensional trees, each leaf node can be taken as a clustering cluster first, the similarity between each two clustering clusters is calculated, for example, the Euclidean distance between the clustering clusters is calculated, and then clustering is performed according to the similarity. The clustering process is repeated until the number of current clustering clusters and the number of clustering clusters before merging meet a preset condition, for example, the number of current clustering clusters is 5% of the number of clustering clusters before merging, and then the clustering is stopped.

[0105] It can be understood that in step B-3, the pixel points in the nearest neighbor K-dimensional tree can also be clustered by using the density-based clustering method, the mean clustering method, or the like.

[0106] 1.3 For step A-3:

[0107] Optionally, the step A-3 "performing convex hull detection on each of the cluster, to obtain a convex hull corresponding to each of the cluster" comprises:

[0108] performing convex hull detection on each of the cluster by using Graham scan method according to the second coordinates of the pixels included in the cluster, to obtain a convex hull corresponding to each of the cluster;

[0109] wherein the second coordinates comprise X-axis coordinate value and Y-axis coordinate value in the first coordinates.

[0110] Therefore, when performing convex hull detection, the pixels are converted from the three-dimensional camera coordinate system to a two-dimensional coordinate system, which is composed of the X-axis and Y-axis of the camera coordinate system, so that Graham scan method is used to perform convex hull detection on each cluster, so that each cluster can obtain a convex hull.

[0111] wherein the process of performing convex hull detection on one of the cluster by using Graham scan method is as follows:

[0112] First, select a pixel point with the smallest Y-axis coordinate value in the cluster as p0, if there are multiple pixel points with the smallest Y-axis coordinate value, select the pixel point with the smallest X-axis coordinate value from the pixel points with the smallest Y-axis coordinate value as p0;

[0113] Second, take p0 as the pole, and sort the pixel points in the cluster according to the polar angle, i.e. the angle between the connecting line of the pixel point to the pole and the polar axis from small to large, wherein the polar axis is the X-axis, and the pixel points with the same polar angle are sorted according to the distance to the pole from small to large, to obtain a target order;

[0114] Third, for example, the target order is (p0, p1, p2…pn), according to the definition of convex hull "given a finite point set on a plane, the convex hull of the point set is the smallest area convex polygon containing all points in the point set", it can be known that p0 and p1 in the target order must belong to the points on the convex hull, so p0 and p1 are pushed into a stack;

[0115] Fourth, determine whether p2 belongs to the points on the convex hull, i.e. according to the first two points p0 and p1 at the top of the stack, if p2 is on the right side of the directed straight line, p2 does not belong to the points on the convex hull, and the element at the top of the stack is popped out; if p2 is on the left side of the directed straight line, p2 belongs to the points on the convex hull, and p2 is pushed into the stack;

[0116] Fifth, repeat the above process to determine whether p3 to pn belong to the points on the convex hull in turn;

[0117] Finally, the elements in the stack are the points on the convex hull.

[0118] Exemplarily, taking a point set (p0, p1, p2, …, p8) as an example, a convex hull of the point set obtained according to the above process is as shown in Figure 3 .

[0119] 1.4 For step A-4:

[0120] When performing plane fitting on the pixel points on the convex hull, a least square method can be used to perform plane fitting to obtain a fitting plane equation and a fitting plane parameter.

[0121] It should be noted that in the embodiments of the present application, only the points on the convex hull are fitted to a plane, thereby saving the calculation time of the plane parameter.

[0122] Optionally, the method further includes the following steps E-1 to E-2:

[0123] Step E-1: Obtain at least one of a target distance and a target area, wherein the target distance includes a distance from each point on the convex hull to the first plane corresponding to the convex hull, and the target area includes an area of a minimum area circumscribed rectangle of each convex hull;

[0124] Step E-2: Delete an abnormal plane in the first plane according to at least one of the target distance and the target area.

[0125] The target distance is used to indicate the positional relationship between the points on the convex hull and the first plane fitted based on the points on the convex hull. If most of the points on a convex hull are far away from the first plane corresponding to the convex hull, the first plane belongs to an abnormal plane.

[0126] The target area is used to indicate the area size of the convex hull corresponding to the first plane. If the area of a convex hull is small, the first plane corresponding to the convex hull belongs to an abnormal plane.

[0127] Therefore, after performing plane fitting on the points on the convex hull to obtain the first plane corresponding to each convex hull, at least one of the target distance and the target area can be further combined to delete the abnormal first plane, thereby further improving the accuracy of plane detection.

[0128] Optionally, in the above step E-2, according to the target distance and the target area, the abnormal plane in the first plane is deleted, including:

[0129] In a case where the number of target distances greater than a third threshold value in the target distance corresponding to the mth convex hull exceeds a target number, or the target area corresponding to the mth convex hull is less than a fourth threshold value, it is determined that the first plane corresponding to the mth convex hull belongs to an abnormal plane, and the abnormal plane is deleted.

[0130] wherein m is an integer from 1 to M.

[0131] Therefore, for any convex hull, if the number of distances greater than the third threshold value among the distances from the points on the convex hull to the first plane corresponding to the convex hull exceeds the target number, or the area of the minimum area circumscribed rectangle of the convex hull is less than the fourth threshold value, the first plane corresponding to the convex hull belongs to an abnormal plane, and thus is deleted.

[0132] It can be understood that, if only the abnormal plane is deleted according to the target distance in the above step E-2, the process can include: in the case that the number of target distances greater than the third threshold value among the target distances corresponding to the mth convex hull exceeds the target number, determining that the first plane corresponding to the mth convex hull belongs to an abnormal first plane, and deleting the abnormal plane; wherein m is an integer from 1 to M.

[0133] Similarly, if only the abnormal plane is deleted according to the target area in the above step E-2, the process can include: in the case that the target area corresponding to the mth convex hull is less than the fourth threshold value, determining that the first plane corresponding to the mth convex hull belongs to an abnormal plane, and deleting the abnormal plane; wherein m is an integer from 1 to M.

[0134] Optionally, the above step 104 of "upsampling the first plane, and adjusting the area boundary of the first plane according to the result of upsampling" includes the following steps K-1 to K-2:

[0135] Step K-1: performing N-layer upsampling on the convex hull corresponding to each first plane to obtain N upsampling boundaries corresponding to each first plane, wherein N is an integer greater than or equal to 1;

[0136] Step K-2: in the case that a second pixel point on the nth upsampling boundary corresponding to the mth first plane is located on the mth first plane, adding the second pixel point to the mth first plane;

[0137] wherein m is an integer from 1 to M, and n is an integer from 1 to N.

[0138] Therefore, in the embodiments of the present application, the boundary of the area of each detected first plane can be adjusted, that is, the convex hull corresponding to each first plane is adjusted, that is, the boundary refinement is performed, so as to improve the accuracy of plane detection. Moreover, by performing multi-layer upsampling on the convex hull corresponding to each first plane, and adding pixel points to the corresponding first plane based on the sampling result of each layer, the final obtained first plane is more matched with the original image collected by the camera, that is, the integrity of the area boundary of the plane is further improved.

[0139] Optionally, the step 105 of detecting the plane of the photographed scene at the second time according to the first plane, a second depth image collected by the camera at the second time in the photographed scene, and a pose change of the camera from the first time to the second time comprises the following steps F1 to F2:

[0140] Step F1: determining a target position of the first plane in the second depth image according to the pose change;

[0141] Step F2: detecting a second plane of the photographed scene at the second time other than the first plane according to the image other than the target position in the second depth image.

[0142] For example Figure 4 As shown in the figure, the first plane 401 is detected at the first time, which is transformed to the position of the third plane 402 in the image detected at the second time, i.e., the next time of the first time, i.e., the first plane 401 and the third plane 402 are the same plane at different times, so the third plane 402 does not need to be detected in the image at the next time.

[0143] Optionally, the method further comprises the following step F-3:

[0144] Step F-3: merging the second plane and the first plane into one plane when the angle between the normal vectors of the second plane and the first plane is less than a fifth threshold value and there is an overlap between the second plane and the first plane.

[0145] For example Figure 4 As shown in the figure, the fourth plane 403 is detected in the image at the second time, and the fourth plane 403 and the second plane 402 have an overlap and the angle between the normal vectors of the fourth plane 403 and the second plane 402 is less than the fifth threshold value, so the fourth plane 403 and the second plane 402 can be merged into one plane.

[0146] Therefore, in the embodiments of the present application, the planes in the detected multiple planes that meet the conditions can also be merged, further improving the accuracy of plane detection.

[0147] As can be seen from the above steps F-1 to F-3, in the embodiments of the present application, the detected planes can also be updated according to the pose change of the camera, thereby avoiding repeated operation of the planes and accelerating the multi-plane fusion.

[0148] The specific implementation of the plane detection method of the embodiments of the present application can be as follows Figure 5 As shown in the figure, the specific implementation is as follows:

[0149] Step 501: Gaussian pyramid down-sampling, that is, adopting a Gaussian pyramid to down-sample a first depth image collected by a camera at a first time in shooting a scene, to obtain a down-sampled image;

[0150] Step 502: 3D point calculation, that is, calculating a first coordinate of each pixel point in the down-sampled image in a camera coordinate system, the specific calculation process of which can be referred to the foregoing description and will not be repeated here;

[0151] Step 503: normal vector calculation, that is, calculating a target normal vector of each pixel point in the down-sampled image in the camera coordinate system according to the first coordinate, the specific calculation process of which can be referred to the foregoing description and will not be repeated here;

[0152] Step 504: horizontal plane pixel point and vertical plane pixel point division, that is, calculating an angle between the target normal vector of each pixel point and a horizontal plane normal vector in the camera coordinate system; in a case where the angle corresponding to the pixel point is less than a first threshold value, the pixel point is divided into a horizontal plane pixel point; in a case where the angle corresponding to the pixel point is greater than a second threshold value, the pixel point is divided into a vertical plane pixel point;

[0153] Step 505: establishing a nearest neighbor K-dimensional tree, that is, establishing a nearest neighbor K-dimensional tree for the horizontal plane pixel points and the vertical plane pixel points respectively, the specific method of constructing the nearest neighbor K-dimensional tree can be referred to the foregoing description and will not be repeated here;

[0154] Step 506: agglomerative hierarchical clustering, that is, performing agglomerative hierarchical clustering on the nearest neighbor K-dimensional tree established for the horizontal plane pixel points and the nearest neighbor K-dimensional tree established for the vertical plane pixel points respectively; the specific clustering process can be referred to the foregoing description and will not be repeated here;

[0155] Step 507: convex hull detection, that is, performing convex hull detection on each clustering cluster obtained in step 506 to obtain a convex hull corresponding to the clustering cluster; the specific convex hull detection process can be referred to the foregoing description and will not be repeated here;

[0156] Step 508: plane fitting, that is, performing plane fitting on the pixel points on the convex hull to obtain a first plane corresponding to each convex hull;

[0157] Step 509: abnormal plane removal, the specific method of which can be referred to the foregoing description (i.e., steps E-1 to E-2) and will not be repeated here;

[0158] Step 510: boundary refinement, the specific method of which can be referred to the foregoing description, i.e., steps K-1 to K-2, and will not be repeated here;

[0159] Step 511: plane updating, that is, detecting the plane of the shooting scene at the second time according to the first plane, the second depth image collected by the camera at the second time in shooting the scene, and the pose change of the camera from the first time to the second time, and performing plane fusion; the specific method of the plane updating can be referred to the foregoing, that is, steps F-1 to F-3, which will not be described here.

[0160] Therefore, the plane detection method provided in the embodiments of the present application can detect the horizontal plane and the vertical plane in the field of view region; the depth image is down-sampled by the Gaussian pyramid, the plane searching is performed in the low-level depth image, the boundary is refined and the plane is updated in the high-level depth image by the multi-layer up-sampling of the detected convex hull, the speed is improved, and the accuracy of the plane searching and calculation is ensured, and the integrity of the region boundary of the plane is improved; and the detected plane is updated according to the pose change of the camera, so that the repeated operation of the plane is avoided, and the multi-plane fusion is accelerated.

[0161] In addition, it should be noted that the plane detection method provided in the embodiments of the present application can be applied to a TOF mobile phone, but the related algorithm can also be applied to other devices capable of providing 3D data and camera pose. For example, a drone.

[0162] The plane detection method provided in the embodiments of the present application can be executed by the plane detection device. The plane detection method executed by the plane detection device is taken as an example in the embodiments of the present application to illustrate the plane detection device provided in the embodiments of the present application.

[0163] In a second aspect, the embodiments of the present application provide a plane detection device, as shown in the following Figure 6 The plane detection device 600 includes the following modules:

[0164] The image acquisition module 601 is configured to acquire a first depth image collected by a camera of an electronic device at a first time in shooting a scene;

[0165] The down-sampling module 602 is configured to down-sample the first depth image to obtain a down-sampled image;

[0166] The first plane detection module 603 is configured to detect a first plane of the scene at the first time according to the down-sampled image;

[0167] The boundary adjustment module 604 is configured to up-sample the first plane, and adjust the region boundary of the first plane according to the up-sampling result;

[0168] The second plane detection module 605 is configured to detect a plane of the shooting scene at a second time according to the first plane, a second depth image collected by the camera at the second time in the shooting scene, and a pose change of the camera from the first time to the second time.

[0169] The second time is later than the first time.

[0170] Optionally, the first plane detection module 603 comprises:

[0171] The first calculation sub-module is configured to calculate a first coordinate of each pixel point in the down-sampling image in a camera coordinate system;

[0172] The clustering processing sub-module is configured to perform clustering processing on each pixel point in the down-sampling image according to the first coordinate, to obtain at least one clustering cluster;

[0173] The convex hull detection sub-module is configured to perform convex hull detection on each clustering cluster, to obtain a convex hull corresponding to each clustering cluster;

[0174] The plane fitting sub-module is configured to perform plane fitting on the pixel points on the convex hull, to obtain a fitting plane corresponding to each convex hull, and determine the fitting plane as the first plane.

[0175] Optionally, the clustering processing sub-module comprises:

[0176] The pixel point division unit is configured to divide each pixel point in the down-sampling image into at least one of horizontal plane pixel points and vertical plane pixel points according to the first coordinate;

[0177] The K-dimensional tree establishment unit is configured to establish a nearest neighbor K-dimensional tree for at least one of the horizontal plane pixel points and the vertical plane pixel points;

[0178] The clustering processing unit is configured to perform clustering processing on the pixel points in the nearest neighbor K-dimensional tree, to obtain at least one clustering cluster.

[0179] Optionally, the pixel point division unit is specifically configured to:

[0180] Calculate a target normal vector of each pixel point in the down-sampling image in the camera coordinate system according to the first coordinate;

[0181] Calculate an included angle between the target normal vector and a horizontal plane normal vector in the camera coordinate system;

[0182] In a case where the included angle corresponding to the pixel point is less than a first threshold value, divide the pixel point into the horizontal plane pixel point;

[0183] In a case where the included angle corresponding to the pixel point is greater than a second threshold, the pixel point is divided into a pixel point of the vertical plane.

[0184] Optionally, the apparatus further includes:

[0185] a parameter acquisition module configured to acquire at least one of a target distance and a target area, wherein the target distance includes a distance from a point on each of the convex hulls to the first plane corresponding to the convex hull, and the target area includes an area of a minimum area circumscribed rectangle of each of the convex hulls;

[0186] an abnormal plane deletion module configured to delete an abnormal plane in the first plane according to at least one of the target distance and the target area.

[0187] The plane detection apparatus in the embodiments of the present applicationapplicationbe an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic deviceapplicationbe a terminal or other devices than a terminal. For example, the electronic deviceapplicationbe a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), andapplicationbe a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, without limitation.

[0188] The plane detection apparatus in the embodiments of the present applicationapplicationbe a device with an operating system. The operating systemapplicationbe an Android operating system, an ios operating system, or other possible operating systems, without limitation.

[0189] The plane detection apparatus provided in the embodiments of the present applicationapplicationimplement the processes of the method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein. Figures 1 to 5

[0190] Optionally, as Figure 7 ​As shown, the electronic device 700 according to the embodiment of the present application includes a processor 701 and a memory 702, and the memory 702 stores programs or instructions executable on the processor 701, which, when executed by the processor 701, implement the steps of the above-mentioned plane detection method embodiment and achieve the same technical effects. To avoid repetition, details are not described herein.

[0191] It should be noted that the electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0192] Figure 8 A hardware structure schematic diagram of an electronic device according to an embodiment of the present application.

[0193] The electronic device 800 includes, but is not limited to, a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, and a processor 810, etc.

[0194] Those skilled in the art can understand that the electronic device 800 can further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 810 through a power management system, so as to realize the functions of power management, such as charging, discharging, and power consumption management, through the power management system. Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the figure, or combine certain components, or different component arrangements, which are not described herein.

[0195] The processor 810 is configured to:

[0196] obtain a first depth image collected by a camera of the electronic device at a first time in a shooting scene;

[0197] down-sample the first depth image to obtain a down-sampled image;

[0198] detect a first plane of the shooting scene at the first time according to the down-sampled image;

[0199] up-sample the first plane, and adjust a region boundary of the first plane according to a result of the up-sampling;

[0200] detect a plane of the shooting scene at a second time according to the first plane, a second depth image collected by the camera at the second time in the shooting scene, and a pose change of the camera from the first time to the second time;

[0201] The second time point is located after the first time point.

[0202] Optionally, the processor 810 detects, according to the down-sampled image, a first plane of the photographed scene at the first time point, comprising:

[0203] calculating a first coordinate of each pixel point in the down-sampled image in a camera coordinate system;

[0204] performing clustering processing on each pixel point in the down-sampled image according to the first coordinate to obtain at least one clustering cluster;

[0205] performing convex hull detection on each clustering cluster to obtain a convex hull corresponding to each clustering cluster;

[0206] performing plane fitting on the pixel points on the convex hull to obtain a fitting plane corresponding to each convex hull, and determining the fitting plane as the first plane.

[0207] Optionally, the processor 810 performs clustering processing on each pixel point in the down-sampled image according to the first coordinate to obtain at least one clustering cluster, comprising:

[0208] dividing each pixel point in the down-sampled image into at least one of a horizontal plane pixel point and a vertical plane pixel point according to the first coordinate;

[0209] establishing a nearest neighbor K-dimensional tree for at least one of the horizontal plane pixel point and the vertical plane pixel point;

[0210] performing clustering processing on the pixel points in the nearest neighbor K-dimensional tree to obtain at least one clustering cluster.

[0211] Optionally, the processor 810 divides each pixel point in the down-sampled image into at least one of a horizontal plane pixel point and a vertical plane pixel point according to the first coordinate, comprising:

[0212] calculating a target normal vector of each pixel point in the down-sampled image in the camera coordinate system according to the first coordinate;

[0213] calculating an included angle between the target normal vector and a horizontal plane normal vector in the camera coordinate system;

[0214] in a case where the included angle corresponding to the pixel point is less than a first threshold value, dividing the pixel point into the horizontal plane pixel point;

[0215] in a case where the included angle corresponding to the pixel point is greater than a second threshold value, dividing the pixel point into the vertical plane pixel point.

[0216] Optionally, the processor 810 is further configured to:

[0217] acquire at least one of a target distance and a target area, wherein the target distance comprises a distance from each of the convex hulls to the first plane corresponding to the convex hull, and the target area comprises an area of a minimum area circumscribed rectangle of each of the convex hulls;

[0218] delete an abnormal plane in the first plane according to at least one of the target distance and the target area.

[0219] The embodiments of the present application can realize the various processes realized by the method embodiments, and achieve the same technical effects. To avoid repetition, details are not described here.

[0220] It should be understood that in the embodiments of the present application, the input unit 804 can include a graphics processor (GPU) 8041 and a microphone 8042. The graphics processor 8041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 806 can include a display panel 8061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 can include a touch detection device and a touch controller. The other input devices 8072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), a trackball, a mouse, an operation lever, and the like, which will not be described here.

[0221] The memory 809 can be used to store software programs and various data. The memory 809 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 809 can include a volatile memory or a non-volatile memory, or the memory 809 can include both a volatile memory and a non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 809 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0222] The processor 810 can include one or more processing units; optionally, the processor 810 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 810.

[0223] The embodiments of the present application also provide a readable storage medium, and the readable storage medium stores programs or instructions, which are executed by a processor to realize various processes of the above-mentioned plane detection method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0224] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0225] The embodiment of the present application further provides a chip, which comprises a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running programs or instructions to realize the processes of the above plane detection method embodiments and achieve the same technical effects. To avoid repetition, details are not described here.

[0226] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system level chip, a system chip, a chip system or a system on chip, etc.

[0227] The embodiment of the present application provides a computer program product, which is stored in a storage medium, and the program product is executed by at least one processor to realize the processes of the above plane detection method embodiments and achieve the same technical effects. To avoid repetition, details are not described here.

[0228] It should be noted that in this document, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to the order of performing the functions as shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in the opposite order, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, the features described with reference to some examples can be combined in other examples.

[0229] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned example methods can be realized by means of software and a necessary general hardware platform, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer software product in essence or in the form of a part that contributes to the prior art, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0230] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative and not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. A planar detection method, characterized by, The method comprises: obtaining a first depth image collected by a camera of an electronic device at a first time in a shooting scene; down-sampling the first depth image to obtain a down-sampled image; detecting a first plane of the shooting scene at the first time according to the down-sampled image; up-sampling the first plane and adjusting a region boundary of the first plane according to a result of the up-sampling; detecting a plane of the shooting scene at a second time according to the first plane, a second depth image collected by the camera at the second time in the shooting scene, and a pose change of the camera from the first time to the second time; wherein the second time is after the first time; the detecting the plane of the shooting scene at the second time according to the first plane, the second depth image collected by the camera at the second time in the shooting scene, and the pose change of the camera from the first time to the second time comprises: determining a target position of the first plane in the second depth image according to the pose change; detecting a second plane of the shooting scene at the second time except the first plane according to the second depth image except the target position; in a case where an included angle between normal vectors of the second plane and the first plane is less than a fifth threshold value and there is an overlap between the second plane and the first plane, merging the second plane and the first plane into one plane.

2. The method of claim 1, wherein, the detecting the first plane of the shooting scene at the first time according to the down-sampled image comprises: calculating first coordinates of each pixel point in the down-sampled image in a camera coordinate system; performing clustering processing on each pixel point in the down-sampled image according to the first coordinates to obtain at least one clustering cluster; performing convex hull detection on each clustering cluster to obtain a convex hull corresponding to each clustering cluster; performing plane fitting on pixel points on the convex hull to obtain a fitting plane corresponding to each convex hull, and determining the fitting plane as the first plane.

3. The method of claim 2, wherein, the performing clustering processing on each pixel point in the down-sampled image according to the first coordinates to obtain at least one clustering cluster comprises: dividing each pixel point in the down-sampled image into at least one of horizontal plane pixel points and vertical plane pixel points according to the first coordinates; establishing a nearest neighbor K-dimensional tree for at least one of the horizontal plane pixel points and the vertical plane pixel points; performing clustering processing on pixel points in the nearest neighbor K-dimensional tree to obtain at least one clustering cluster.

4. The method of claim 3, wherein, the dividing each pixel point in the down-sampled image into at least one of horizontal plane pixel points and vertical plane pixel points according to the first coordinates comprises: calculating target normal vectors of each pixel point in the down-sampled image in the camera coordinate system according to the first coordinates; calculating included angles between the target normal vectors and horizontal plane normal vectors in the camera coordinate system; in a case where the included angles corresponding to the pixel points are less than a first threshold value, dividing the pixel points into the horizontal plane pixel points; In a case where the included angle corresponding to the pixel point is greater than a second threshold, the pixel point is divided into a pixel point of the vertical plane.

5. The method of claim 2, wherein, The method further includes: acquiring at least one of a target distance and a target area, wherein the target distance includes a distance from each point on the convex hull to the first plane corresponding to the convex hull, and the target area includes an area of a minimum area circumscribed rectangle of each convex hull; according to at least one of the target distance and the target area, deleting an abnormal plane in the first plane.

6. A flatness detecting device characterized by comprising: The device includes: an image acquisition module, configured to acquire a first depth image collected by a camera of an electronic device at a first time in a shooting scene; a down-sampling module, configured to down-sample the first depth image to obtain a down-sampled image; a first plane detection module, configured to detect a first plane of the shooting scene at the first time according to the down-sampled image; a boundary adjustment module, configured to up-sample the first plane, and adjust a region boundary of the first plane according to a result of the up-sampling; a second plane detection module, configured to detect a plane of the shooting scene at a second time according to the first plane, a second depth image collected by the camera at the second time in the shooting scene, and a pose change of the camera from the first time to the second time; wherein the second time is after the first time; the second plane detection module is specifically configured to determine a target position of the first plane in the second depth image according to the pose change; detect a second plane of the shooting scene at the second time, except the first plane, according to an image of the second depth image except the target position; and in a case where an included angle between the second plane and a normal vector of the first plane is less than a fifth threshold, and the second plane and the first plane overlap, merge the second plane and the first plane into one plane.

7. The apparatus of claim 6, wherein, The first plane detection module includes: a first calculation sub-module, configured to calculate a first coordinate of each pixel point in the down-sampled image in a camera coordinate system; a clustering processing sub-module, configured to perform clustering processing on each pixel point in the down-sampled image according to the first coordinate, to obtain at least one clustering cluster; a convex hull detection sub-module, configured to perform convex hull detection on each clustering cluster, to obtain a convex hull corresponding to each clustering cluster; a plane fitting sub-module, configured to perform plane fitting on pixel points on the convex hull, to obtain a fitting plane corresponding to each convex hull, and determine the fitting plane as the first plane.

8. The apparatus of claim 7, wherein, The clustering processing sub-module includes: a pixel point division unit, configured to divide each pixel point in the down-sampled image into at least one of a horizontal plane pixel point and a vertical plane pixel point according to the first coordinate; a K-dimensional tree establishment unit, configured to establish a nearest neighbor K-dimensional tree for at least one of the horizontal plane pixel point and the vertical plane pixel point; a clustering processing unit, configured to perform clustering processing on pixel points in the nearest neighbor K-dimensional tree, to obtain at least one clustering cluster.

9. The apparatus of claim 8, wherein, The pixel point division unit is specifically configured to: According to the first coordinates, a target normal vector of each pixel point in the down-sampled image in the camera coordinate system is calculated; An angle between the target normal vector and a horizontal plane normal vector in the camera coordinate system is calculated; In a case where the angle corresponding to the pixel point is less than a first threshold value, the pixel point is divided into a pixel point on the horizontal plane; In a case where the angle corresponding to the pixel point is greater than a second threshold value, the pixel point is divided into a pixel point on the vertical plane.

10. The apparatus of claim 7, wherein, The device further comprises: a parameter acquisition module configured to acquire at least one of a target distance and a target area, wherein the target distance comprises a distance from a point on each of the convex hulls to the first plane corresponding to the convex hull, and the target area comprises an area of a minimum area circumscribed rectangle of each of the convex hulls; an abnormal plane deletion module configured to delete an abnormal plane in the first plane according to at least one of the target distance and the target area.

Images