Plane estimation method and device, electronic equipment and storage medium
By determining the homography matrix and edge images in images at different perspectives, calculating the position deviation of projected pixel points and constructing optimization problems, the problem of poor plane estimation accuracy in the prior art is solved, and more efficient and accurate plane estimation is achieved.
Patent Information
- Application Number
- CN202311706239.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-12
- Publication Date
- 2025-06-13
AI Technical Summary
The existing image-based plane estimation algorithm has high computational complexity and does not involve coplanar constraints during depth estimation, resulting in poor plane estimation accuracy, especially in weak texture areas.
By acquiring two images at different perspectives, the homography transformation matrix of the target space plane in the two images is determined, the edge image is extracted, the position deviation of the projected pixel points is calculated, and the optimization problem is constructed with the goal of minimizing the position deviation, and the optimal plane parameters are solved.
It reduces the computational complexity, improves the accuracy of plane estimation, avoids the problem of non-coplanarity of depth estimation results, and is suitable for plane estimation in weak texture areas.
Smart Images
Figure CN120147412A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a plane estimation method, apparatus, electronic device, and computer storage medium. Background Art
[0002] For image-based plane estimation, its main task is to analyze an image containing a spatial plane to obtain the plane parameters of the above-mentioned spatial plane. In practical applications, the spatial plane may refer to the surface of an object, such as: a wall surface, a ground surface, a tabletop, etc.
[0003] Existing image-based plane estimation algorithms usually first perform depth estimation and then perform plane fitting on the plane region. Depth estimation usually requires estimating the depth of each pixel, with a relatively high computational complexity; moreover, the depth estimation process does not involve coplanarity constraints. Therefore, the result of depth estimation is not coplanar. In addition, for weakly textured plane regions (such as solid-color wall surfaces, tabletops, etc.), the depth estimation accuracy will be greatly affected, which will in turn lead to poor accuracy of the final plane estimation result. Summary of the Invention
[0004] In view of this, embodiments of the present disclosure provide a plane estimation solution.
[0005] According to a first aspect of the embodiments of the present disclosure, a plane estimation method is provided, including:
[0006] Obtain a first image and a second image, both of which contain a target spatial plane, and the shooting perspectives of the first image and the second image are different;
[0007] Determine a homography transformation matrix of the target spatial plane in the first image and the second image; wherein, the homography transformation matrix is related to the plane parameters of the target spatial plane;
[0008] Extract a first edge image of the target spatial plane in the first image and a second edge image of the target spatial plane in the second image;
[0009] Based on the homography transformation matrix, determine the projected pixel points corresponding to the pixels of the first edge in the first edge image in the second edge image;
[0010] Determine the position deviation between the projected pixel points and the second edge in the second edge image; wherein, the position deviation is a function of the plane parameters of the target spatial plane;
[0011] Construct an optimization problem with the minimum position deviation as the objective function, and solve for the optimal plane parameters.
[0012] According to a second aspect of the embodiments of the present disclosure, a plane estimation device is provided, including:
[0013] An image acquisition module, configured to acquire a first image and a second image, both of which include a target spatial plane, and the shooting perspectives of the first image and the second image are different;
[0014] A homography transformation matrix determination module, configured to determine the homography transformation matrix of the target spatial plane between the first image and the second image; wherein, the homography transformation matrix is related to the plane parameters of the target spatial plane;
[0015] An edge image extraction module, configured to extract a first edge image of the target spatial plane in the first image and a second edge image of the target spatial plane in the second image;
[0016] A projected pixel point determination module, configured to determine the projected pixel points corresponding to the pixels of the first edge in the first edge image in the second edge image based on the homography transformation matrix;
[0017] A position deviation determination module, configured to determine the position deviation between the projected pixel points and the second edge in the second edge image; wherein, the position deviation is a function with the plane parameters of the target spatial plane as independent variables;
[0018] A plane parameter solving module, configured to construct an optimization problem with the minimum position deviation as the objective function and solve for the optimal plane parameters.
[0019] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, including: a processor; and a memory storing a program; wherein, the program includes instructions that, when executed by the processor, cause the processor to execute the plane estimation method according to the first aspect above.
[0020] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, it implements the plane estimation method according to the first aspect above. Description of the Drawings
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.
[0022] Figure 1Shows a flowchart of steps of a plane estimation method provided by an embodiment of the present disclosure;
[0023] Figure 2 Shows a schematic diagram of obtaining corresponding images by photographing a target space plane from different perspectives;
[0024] Figure 3 Is a schematic diagram of the projection pixel points corresponding to the pixel points of the first edge in the first edge image in the second edge image;
[0025] Figure 4 Is a structural block diagram of a plane estimation device provided by an embodiment of the present disclosure;
[0026] Figure 5 Is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. Specific embodiments
[0027] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure, the following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art shall fall within the protection scope of the embodiments of the present disclosure.
[0028] The following further illustrates the specific implementation of the embodiments of the present disclosure in conjunction with the accompanying drawings of the embodiments of the present disclosure.
[0029] The solution of the embodiment of the present disclosure can be applied to any suitable electronic device with data processing capabilities. For example, the electronic device can be used to obtain a first image and a second image including a target space plane from different shooting perspectives; based on the homography transformation matrix of the target space plane between the first image and the second image, determine the position deviation between the first edge in the first image and the second edge in the second image, where the above position deviation is a function of the plane parameters of the target space plane; construct an optimization problem based on minimizing the above position deviation, and solve for the optimal plane parameters (that is, complete the plane estimation of the target space plane). After that, based on the solved optimal plane parameters, three-dimensional reconstruction can be performed to achieve virtual-real interaction. For example, it can be used to achieve virtual-real interaction in application scenarios such as Augmented Reality (AR) and Virtual Reality (VR). It can also determine the position of the anchor point based on the solved optimal plane parameters, and perform image rendering of virtual objects at the anchor point position.
[0030] The present disclosure aims to propose a plane estimation method, which mainly includes the following steps: obtaining a first image and a second image, both of which contain a target space plane, and the shooting perspectives of the first image and the second image are different; determining the homography transformation matrix of the target space plane in the first image and the second image; extracting a first edge image of the target space plane in the first image and a second edge image of the target space plane in the second image; based on the homography transformation matrix, determining the corresponding projected pixel points in the second edge image for each pixel point of the first edge in the first edge image; determining the position deviation between the projected pixel points and the second edge in the second edge image; wherein the position deviation is a function of the plane parameters of the target space plane; constructing an optimization problem with minimizing the position deviation as the objective function, and solving for the optimal plane parameters.
[0031] In an embodiment of the present disclosure, by calculating the position deviation between the plane edges in the first image and the plane edges in the second image, the plane estimation problem is transformed into an optimization problem, and finally the optimal plane parameters are obtained. The computational complexity is low, and the accuracy of the finally obtained plane estimation result is higher. In addition, using the obtained optimal plane parameters for plane fitting can avoid the problem of non-coplanarity of the depth estimation results, and thus is beneficial to three-dimensional reconstruction based on the optimal plane parameters.
[0032] Figure 1 The flowchart of the steps of the plane estimation method provided by some embodiments of the present disclosure is shown. As Figure 1 shown, the plane estimation method may include the following steps:
[0033] Step 102, obtaining a first image and a second image, both of which contain a target space plane, and the shooting perspectives of the first image and the second image are different.
[0034] The target space plane can be custom-set according to actual requests. For example, it can be a desktop, a wall surface, etc.
[0035] The first image and the second image can be acquired by an image acquisition device (such as a camera, etc.). Optionally, the device for acquiring the spatial plane image can be a monocular camera or a binocular camera.
[0036] When using a monocular camera for image acquisition, images of the target space plane can be acquired from different perspectives at different times; when using a binocular camera for image acquisition, the relative pose between the two cameras can be pre-calibrated, and then, at the same time, images containing the target space plane are acquired by the left and right cameras simultaneously.
[0037] Step 104, determining the homography transformation matrix of the target space plane in the first image and the second image; wherein the homography transformation matrix is related to the plane parameters of the target space plane.
[0038] The homography matrix is used to describe the transformation relationship of an object between two planes. In the embodiments of the present disclosure, the homography matrix of the target space plane between the first image and the second image is used to describe the transformation relationship of the points on the target space plane between the first image and the second image.
[0039] The plane parameters can be expressed as: pane=(normal, d), where normal is the normal vector and d is the intercept.
[0040] See Figure 2 , Figure 2 which shows schematic diagrams of corresponding images obtained by photographing the target space plane from different perspectives in some embodiments of the present disclosure. Among them, for the target space plane P, the first image T1 is obtained by photographing at the position O1 from the left perspective, and the second image T2 is obtained by photographing at the position O2 from the right perspective. The first image T1 contains the plane region P1 corresponding to the target space plane P, and the second image T1 contains the plane region P2 corresponding to the target space plane P. Moreover, for a vertex A on the edge of the target space plane P, the first image T1 contains the point A1 corresponding to the point A, and the second image T2 contains the point A2 corresponding to the point A. The homography matrix of the target space plane between the first image and the second image can be used to describe the transformation relationship (mapping relationship) between the coordinates of the point A1 in the first image and the coordinates of the point A2 in the second image.
[0041] The homography matrix ( Figure 2 H in) corresponding to the target space plane is related to the plane parameters of the target space plane. The homography matrix of the target space plane between the first image and the second image can be calculated by the following formula 1:
[0042]
[0043]
[0044] where H is the homography matrix of the target space plane between the first image and the second image; K 1 is the internal parameter of the first image acquisition device for photographing the first image; K 2The internal parameters of the second image acquisition device for capturing the second image; R represents the relative rotation matrix between the first image acquisition device and the second image acquisition device; t represents the relative translation matrix between the first image acquisition device and the second image acquisition device; n is the normal vector of the spatial plane, and d is the intercept of the target spatial plane; normal.x is the component of the normal vector normal of the target spatial plane in the x-axis direction; normal.y is the component of the normal vector of the target spatial plane in the y-axis direction; normal.z is the component of the normal vector of the target spatial plane in the z-axis direction.
[0045] Step 106: Extract the first edge image of the target spatial plane in the first image and the second edge image of the target spatial plane in the second image.
[0046] Optionally, image segmentation can be performed on the first image and the second image respectively to obtain the first segmentation image corresponding to the first image and the second segmentation image corresponding to the second image; then, edge extraction is performed on the first segmentation image and the second segmentation image respectively to obtain the first edge image of the target spatial plane in the first image and the second edge image of the target spatial plane in the second image.
[0047] In the embodiments of the present disclosure, the specific segmentation method used for image segmentation is not limited and can be custom-set according to the actual situation. Similarly, in the embodiments of the present disclosure, the specific extraction method used for edge extraction is not limited and can also be custom-set according to the actual situation.
[0048] Step 108: Based on the homography transformation matrix, determine the projected pixel points corresponding to each pixel point of the first edge in the first edge image in the second edge image.
[0049] In the embodiments of the present disclosure, based on the homography transformation matrix, the projected pixel points corresponding to all pixel points in the first edge in the second edge image can be calculated, or the projected pixel points corresponding to some pixel points in the first edge in the second edge image can be calculated. When calculating the projected pixel points corresponding to some pixel points in the second edge image, a preset selection principle can be used to select some pixel points from the first edge. In the embodiments of the present disclosure, the specific pixel point selection principle is not limited and can be custom-set according to the actual situation.
[0050] Step 110: Determine the position deviation between the projected pixel points and the second edge in the second edge image; where the position deviation is a function of the plane parameters of the target spatial plane.
[0051] Optionally, the positional deviation between the projection pixel points and the second edge in the second edge image is the shortest distance from the projection pixel points in the second edge image to the second edge.
[0052] Since the homography transformation matrix determined in step 104 is related to the plane parameters of the target space plane, and in step 108, the projection pixel points are determined based on the homography transformation matrix, therefore, in step 110, the positional deviation between the determined projection pixel points and the second edge in the second edge image is also related to the plane parameters of the target space plane. The above-mentioned positional deviation determined in step 110 is a function with the plane parameters of the target space plane as the independent variable.
[0053] Step 112: Construct an optimization problem with the minimization of the positional deviation as the objective function, and solve for the optimal plane parameters.
[0054] Before solving for the optimal plane parameters, since the plane parameters are not accurate enough, therefore, the projection pixel points corresponding to the pixel points of the first edge in the first edge image obtained through the above steps 104 - 108 may not be on the second edge in the second edge image. Refer to Figure 3 , Figure 3 is a schematic diagram of the projection pixel points corresponding to the pixel points of the first edge in the first edge image in the second edge image. For Figure 3 the pixel point B1 of the first edge in the first edge image in, using the homography transformation matrix H obtained with inaccurate plane parameters, the projection pixel point corresponding to B1 in the second edge image is determined to be B2. From Figure 3 it can be seen that: B2 is not on the second edge in the second edge image, but is outside the second edge. It can be understood that due to the inaccurate plane parameters, B2 may also appear inside the second edge.
[0055] If the plane parameters are accurate enough, the projection pixel points corresponding to the pixel points of the first edge in the first edge image obtained through the above steps 104 - 108 should be on the second edge in the second edge image. Therefore, it can be seen that: the accuracy of the plane parameters will affect the positional deviation between the above-mentioned projection pixel points and the second edge in the second edge image. Among them, the higher the accuracy, the smaller the above-mentioned positional deviation; conversely, when the above-mentioned positional deviation is the smallest, it indicates that the accuracy of the plane parameters is also the highest. For the above reasons, after determining the positional deviation between the projection pixel points and the second edge in the embodiments of the present disclosure, by constructing an optimization problem with the minimization of the positional deviation as the objective function, the optimal plane parameters can be solved.
[0056] In the embodiments of the present disclosure, by calculating the position deviation between the planar edge in the first image and the planar edge in the second image, the planar estimation problem is transformed into an optimization problem, and finally the optimal planar parameters of the target spatial plane are obtained. The calculation complexity is low, and the accuracy of the finally obtained planar estimation result is high.
[0057] In some alternative embodiments, the process of determining the position deviation between the projected pixel point and the second edge in step 110 above may include:
[0058] Calculating the distance transformation image dt corresponding to the second edge image;
[0059] Obtaining the minimum distance from the projected pixel point to the second edge according to the distance transformation image dt, and determining the position deviation based on the minimum distance.
[0060] After extracting the second edge image in step 106 above, the closest distance from each pixel point in the second edge image to the second edge can be calculated, so as to obtain the distance transformation image dt corresponding to the second edge image. Among them, the value dt(u, v) at the position of each pixel point in the distance transformation image indicates the closest distance from the pixel point at the position (u, v) to the second edge. The closest distance from each pixel point in the second edge image to the second edge may be the distance from each pixel in the second edge image to the closest edge pixel.
[0061] After obtaining the distance transformation image corresponding to the second edge image, since the projected pixel points corresponding to the pixel points of the first edge in the first edge image have been obtained in step 108 above, therefore, based on the distance transformation image dt, the minimum distance from the projected pixel point to the second edge can be obtained, and the position deviation e(p i ) can be determined based on the minimum distance:
[0062] e(p i ) = dt(u, v) (Equation 3)
[0063] p 2 = H * p 1 (Equation 4)
[0064]
[0065]
[0066] where (u, v) is the image coordinate of the pixel point on the planar region P2 in the second image T2; p 1 is the homogeneous coordinate corresponding to the pixel point on the planar region P1 in the first image T1, p 1 = (p 1 .x, p 1 .y, p 1.z), p 2 are the coordinates of the pixel points on the plane region P2 in the second image T2, p 2 =(p 2 .x, p 2 .y, p 2 .z).
[0067] In the embodiments of the present disclosure, there is no limitation on the specific manner of determining the position deviation based on the above minimum distance, and it can be custom-set according to the actual situation. For example: the sum of the minimum distances from each projection pixel point to the second edge can be determined as the above position deviation, or the average value of the minimum distances from each projection pixel point to the second edge can be determined as the above position deviation, or the variance of the minimum distances from each projection pixel point to the second edge can be determined as the above position deviation, and so on.
[0068] In some alternative embodiments, before the step of constructing an optimization problem with the position deviation minimization as the objective function and solving for the optimal plane parameters, the method further includes:
[0069] Determine the pixel value deviation e between the projection pixel point p and the corresponding pixel point pi on the first edge p (p i ):
[0070] e p (p i ) = I 2 (p) - I 1 (p i ) (Equation 7)
[0071] wherein, the pixel value deviation e p (p i ) is a function with the plane parameters of the target space plane as the independent variable;
[0072] Constructing an optimization problem with the position deviation minimization as the objective function and solving for the optimal plane parameters includes: constructing an optimization problem with the sum of the position deviation and the pixel value deviation minimized as the objective function and solving for the optimal plane parameters.
[0073] The position deviation determined in step 110, and the pixel value deviation between the projected pixel points and the pixel points of the first edge determined in the above process, are both functions of the plane parameters of the target space plane. Since the plane parameters of the target space plane cannot be accurately obtained, the position deviation determined in step 110 represents the position deviation of the boundary of the target space plane at different viewing angles, while the pixel value deviation between the projected pixel points and the pixel points of the first edge determined in the above process represents the imaging deviation of the target space plane at different viewing angles. For example, when the first image and the second image obtained in step 102 above are both grayscale images, the pixel value of each pixel point in the image is a grayscale value, and the above pixel value deviation is the difference in grayscale values. When constructing the optimization problem, both the above position deviation and pixel value deviation are considered, and the sum of the position deviation and pixel value deviation is minimized as the objective function to construct the optimization problem, so that the finally solved optimal plane parameters can be more accurate.
[0074] In some alternative embodiments, before the step of solving the optimal plane parameters, the method further includes: obtaining the initial plane parameters of the target space plane;
[0075] Solving the optimal plane parameters includes: using the initial plane parameters as the initial value of the optimization problem, and iterating the plane parameters to obtain the optimal plane parameters.
[0076] When solving the optimization problem, an initial value can be set for the plane parameters to be optimized first, which can improve the solution speed of the optimization problem and the efficiency of plane estimation. In the examples of the present disclosure, there is no limitation on the specific method for setting the initial plane parameters, which can be custom-set according to the actual situation.
[0077] Before optimization, the plane parameters are unknown, and the given initial plane parameters are usually not accurate. Therefore, the above deviations (position deviation, the latter, position deviation and pixel value deviation) are relatively large. The optimal plane parameters are obtained by minimizing the above deviations through an optimization method.
[0078] It can be understood that the solution process of the optimization problem is an iterative process. According to the current residual and derivative, the update amount of the plane parameters of the target space plane is calculated, and the plane parameters are updated. According to the updated plane parameters, a new position deviation (or a new position deviation and a new pixel value deviation) is calculated until the position deviation (or the sum of the position deviation and the pixel value deviation) reaches the minimum. Then, the plane parameters that minimize the position deviation (or the sum of the position deviation and the pixel value deviation) are considered as the optimal plane parameters.
[0079] In some alternative embodiments, the initial plane parameters include an initial normal vector; obtaining the initial plane parameters of the target space plane may include:
[0080] Perform semantic segmentation on at least one of the first image and the second image to obtain a semantic segmentation result;
[0081] Determine the plane type of the target space plane according to the semantic segmentation result;
[0082] Determine the initial normal vector according to the plane type.
[0083] Optionally, the plane type of the target space plane can be determined by means of semantic segmentation, and then the initial normal vector can be determined according to the plane type. For example: for common semantic planes (such as tables, floors, walls, etc.), common sense priors can be used. For example, the initial normal vector of plane types such as the floor and the tabletop can be the opposite direction of gravity, while the initial normal vector of plane types such as the wall can be perpendicular to the direction of gravity.
[0084] In the above method, prior knowledge is used to set the initial normal vector of the target space plane according to the type of the plane, so that the finally determined initial normal vector can be closer to the true normal vector of the target space plane. Furthermore, based on the above initial normal vector closer to the true normal vector for optimization solution, the optimal normal vector can be obtained more quickly, improving the efficiency of plane estimation.
[0085] In some alternative embodiments, the initial plane parameters include an initial intercept; obtaining the initial plane parameters of the target space plane may include:
[0086] Sample a plurality of candidate initial intercepts within a preset value range of the initial intercept to form a plurality of candidate initial plane parameters;
[0087] Based on each of the plurality of candidate initial plane parameters, calculate the total position deviation between the projected pixel points and the second edge in the second edge image, and select the candidate initial plane parameter corresponding to the minimum total position deviation as the initial plane parameter.
[0088] In the above method, the final initial intercept is selected from a plurality of candidate initial intercepts based on the magnitude of the total position deviation. Since the total position deviation corresponding to the selected final initial intercept is the smallest, it indicates that the selected final initial intercept is closer to the true intercept of the target space plane. Furthermore, based on the above initial intercept closer to the true intercept for optimization solution, the optimal intercept can be obtained more quickly, improving the efficiency of plane estimation.
[0089] In some alternative embodiments, sampling within the value range of the initial intercept to obtain a plurality of candidate initial intercepts may include:
[0090] Perform uniform sampling within the value range of the initial intercept at a preset interval to obtain a plurality of candidate initial intercepts.
[0091] In some alternative embodiments, the optimization problem may be a gradient-based optimization problem, and the partial derivative of the position deviation with respect to the plane parameters is used as the gradient of the optimization problem.
[0092] In the embodiments of the present disclosure, no specific method is limited for solving the gradient-based optimization problem, and a suitable method can be selected according to the actual situation. For example: The Gauss-Newton or Levenberg-Marquardt method can be used for optimization. After the optimization converges, the optimal plane parameters can be obtained.
[0093] Assume that for a pixel point p on the first edge in the first edge image i , its corresponding projected pixel point in the second edge image is p, then p i and p satisfy the following formula:
[0094]
[0095] Where is the representation of the normal vector n in the spherical coordinate system, where is the azimuth angle between the projection line of the normal vector on the xy plane and the positive x axis, and θ is the zenith angle between the normal vector and the positive z axis; is the homography transformation matrix related to θ and the intercept d; The normal vector normal of the target space plane P is represented in spherical coordinates as:
[0096]
[0097] Based on the above formula, the partial derivatives of the above position deviation e(p i ) with respect to the plane parameters φ, θ, d of the target space plane are as follows:
[0098]
[0099] Where dx is the partial derivative of the above dt with respect to the x axis, and dy is the partial derivative of the above dt with respect to the y axis.
[0100] In some embodiments of the present disclosure, the partial derivatives of the above pixel value deviation e p (p i ) with respect to the plane parameters of the target space plane are as follows:
[0101]
[0102] Next, the partial derivative of u with respect to the plane parameters (φ, θ, d) and, the partial derivative of v with respect to the plane parameters (φ, θ, d) The derivation process is explained as follows:
[0103] Based on Equation 1 and Equations 4 - 6, the following equation can be obtained:
[0104]
[0105]
[0106]
[0107]
[0108]
[0109]
[0110] Based on the above Equation 9, the following can be obtained:
[0111]
[0112]
[0113] Among them, Identity33 represents a 3×3 identity matrix.
[0114] Furthermore, the following can be obtained:
[0115]
[0116] And,
[0117]
[0118]
[0119] So far, the partial derivatives of u with respect to the plane parameters (φ, θ, d) can be obtained And, the partial derivatives of v with respect to the plane parameters (φ, θ, d) Substituting them into the above Equations 10 and 11, the partial derivatives of the position deviation e(pi) with respect to the plane parameters φ, θ, d of the target space plane can be obtained, and the pixel value deviation e p (p i ) with respect to the plane parameters φ, θ, d of the target space plane. Furthermore, the optimal plane parameters of the target space plane are obtained by solving the gradient-based optimization problem.
[0120] Referring to Figure 4 , Figure 4 is the structural block diagram of the plane estimation device provided according to the embodiments of the present disclosure. The plane estimation device provided by the embodiments of the present disclosure includes:
[0121] An image acquisition module 402, configured to acquire a first image and a second image. Both the first image and the second image contain a target space plane, and the shooting perspectives of the first image and the second image are different;
[0122] A homography transformation matrix determination module 404, configured to determine the homography transformation matrix of the target space plane between the first image and the second image; wherein, the homography transformation matrix is related to the plane parameters of the target space plane;
[0123] An edge image extraction module 406, configured to extract a first edge image of the target space plane in the first image and a second edge image of the target space plane in the second image;
[0124] A projected pixel point determination module 408, configured to determine, based on the homography transformation matrix, the projected pixel points corresponding to the pixels of the first edge in the first edge image in the second edge image;
[0125] A position deviation determination module 410, configured to determine the position deviation between the projected pixel points and the second edge in the second edge image; wherein, the position deviation is a function of the plane parameters of the target space plane;
[0126] A plane parameter solving module 412, configured to construct an optimization problem with the minimum position deviation as the objective function, and solve for the optimal plane parameters.
[0127] In some alternative embodiments, the position deviation determination module 410 is specifically configured to:
[0128] Calculate the distance transformation image corresponding to the second edge image;
[0129] Obtain the minimum distance from the projected pixel points to the second edge according to the distance transformation image, and determine the position deviation based on the minimum distance.
[0130] In some alternative embodiments, the plane estimation device further includes:
[0131] A pixel value deviation determination module, configured to determine the pixel value deviation between the projected pixel points and the pixel points of the corresponding first edge before the step of constructing an optimization problem with the minimum position deviation as the objective function and solving for the optimal plane parameters; wherein, the pixel value deviation is a function of the plane parameters of the target space plane;
[0132] The plane parameter solving module 412 is specifically configured to: construct an optimization problem with the sum of the position deviation and the pixel value deviation minimized as the objective function, and solve for the optimal plane parameters.
[0133] In some alternative embodiments, before performing the step of solving for the optimal plane parameters, the homography transformation matrix determination module 404 is further configured to: obtain the initial plane parameters of the target space plane; the homography transformation matrix is determined based on the initial plane parameters.
[0134] When performing the step of solving for the optimal plane parameters, the plane parameter solving module 412 is specifically configured to: use the initial plane parameters as the initial value of the optimization problem, and iterate on the plane parameters to obtain the optimal plane parameters.
[0135] In some alternative embodiments, the initial plane parameters include an initial normal vector; when performing the step of obtaining the initial plane parameters of the target space plane, the homography transformation matrix determination module 404 is specifically configured to: perform semantic segmentation on at least one of the first image and the second image to obtain a semantic segmentation result; determine the plane type of the target space plane according to the semantic segmentation result; and determine the initial normal vector according to the plane type.
[0136] In some alternative embodiments, the initial plane parameters include an initial intercept; when performing the step of obtaining the initial plane parameters of the target space plane, the homography transformation matrix determination module 404 is specifically configured to: sample within a preset value range of the initial intercept to obtain a plurality of candidate initial intercepts, and form a plurality of candidate initial plane parameters; calculate the total position deviation between the projection pixels and the second edge in the second edge image based on each of the plurality of candidate initial plane parameters, and select the candidate initial plane parameter corresponding to the minimum total position deviation as the initial plane parameter.
[0137] In some alternative embodiments, when performing the step of sampling within a preset value range of the initial intercept to obtain a plurality of candidate initial intercepts, the homography transformation matrix determination module 404 is specifically configured to: uniformly sample within the value range of the initial intercept at a preset interval to obtain a plurality of candidate initial intercepts.
[0138] In some alternative embodiments, the optimization problem is a gradient-based optimization problem, and the partial derivative of the position deviation with respect to the plane parameters is used as the gradient of the optimization problem.
[0139] The plane estimation device of this embodiment is used to implement the corresponding plane estimation methods in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated herein. In addition, the functional implementation of each module in the plane estimation device of this embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will not be elaborated herein either.
[0140] See Figure 5 , Figure 5The figure is a schematic structural diagram of an electronic device provided according to an embodiment of the present disclosure. The specific implementation of the electronic device is not limited in the specific embodiments of the present disclosure.
[0141] As Figure 5 shown, the electronic device may include: a processor 502, a communications interface 504, a memory 506, and a communication bus 508.
[0142] Among them:
[0143] The processor 502, the communications interface 504, and the memory 506 communicate with each other through the communication bus 508.
[0144] The communications interface 504 is used to communicate with other electronic devices or servers.
[0145] The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the above-mentioned embodiment of the plane estimation method.
[0146] Specifically, the program 510 may include program code, and the program code includes computer operation instructions.
[0147] The processor 502 may be a CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present disclosure. One or more processors included in the intelligent device may be of the same type of processor, such as one or more CPUs; or may be of different types of processors, such as one or more CPUs and one or more ASICs.
[0148] The memory 506 is used to store the program 510. The memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.
[0149] The program 510 may include multiple computer instructions. Specifically, the program 510 may cause the processor 502 to execute the operations corresponding to the plane estimation method described in any one of the foregoing multiple method embodiments through the multiple computer instructions.
[0150] For the specific implementation of each step in Program 510, reference may be made to the corresponding descriptions in the corresponding steps and units in the foregoing method embodiments, and they have corresponding beneficial effects, which will not be elaborated here. Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the devices and modules described above can refer to the corresponding process descriptions in the foregoing method embodiments, which will not be elaborated here.
[0151] The embodiments of the present disclosure also provide a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the method described in any one of the foregoing multiple method embodiments. The computer storage medium includes but is not limited to: Compact Disc Read-Only Memory (CD-ROM), Random Access Memory (RAM), floppy disk, hard disk, magneto-optical disk, etc.
[0152] It should be noted that according to the needs of implementation, the various components / steps described in the embodiments of the present disclosure can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present disclosure.
[0153] The method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or be implemented as software or computer code that can be stored in a recording medium (such as CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or be implemented as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium and to be stored in a local recording medium and downloaded through a network, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (such as Random Access Memory (RAM), Read-Only Memory (ROM), flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, it implements the method described herein. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for implementing the method shown herein.
[0154] Those of ordinary skill in the art can realize that the units and method steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present disclosure.
[0155] The above embodiments are only used to illustrate the embodiments of the present disclosure, rather than limiting the embodiments of the present disclosure. Those of ordinary skill in the relevant technical field can also make various changes and modifications without departing from the spirit and scope of the embodiments of the present disclosure. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present disclosure. The patent protection scope of the embodiments of the present disclosure shall be defined by the claims.
Claims
1. A method for plane estimation, comprising: obtaining a first image and a second image, both of which contain a target spatial plane, and the shooting perspectives of the first image and the second image are different; determining a homography transformation matrix of the target spatial plane in the first image and the second image; wherein, the homography transformation matrix is related to the plane parameters of the target spatial plane; extracting a first edge image of the target spatial plane in the first image and a second edge image of the target spatial plane in the second image; determining, based on the homography transformation matrix, the projected pixel points corresponding to each pixel point of the first edge in the first edge image in the second edge image; determining the position deviation between the projected pixel points and the second edge in the second edge image; wherein, the position deviation is a function with the plane parameters of the target spatial plane as independent variables; constructing an optimization problem with the minimization of the position deviation as the objective function, and solving for the optimal plane parameters.
2. The method according to claim 1, wherein, the determining the position deviation between the projected pixel points and the second edge includes: calculating a distance transformation image corresponding to the second edge image; obtaining the minimum distance from the projected pixel points to the second edge according to the distance transformation image, and determining the position deviation based on the minimum distance.
3. The method according to claim 1 or 2, wherein, before the step of constructing an optimization problem with the minimization of the position deviation as the objective function and solving for the optimal plane parameters, the method further includes: determining the pixel value deviation between the projected pixel points and the corresponding pixel points of the first edge; wherein, the pixel value deviation is a function with the plane parameters of the target spatial plane as independent variables; the constructing an optimization problem with the minimization of the position deviation as the objective function and solving for the optimal plane parameters includes: constructing an optimization problem with the minimization of the sum of the position deviation and the pixel value deviation as the objective function, and solving for the optimal plane parameters.
4. The method according to any one of claims 1-3, wherein, before the step of solving for the optimal plane parameters, the method further includes: obtaining the initial plane parameters of the target spatial plane; the solving for the optimal plane parameters includes: using the initial plane parameters as the initial value of the optimization problem, and iterating the plane parameters to obtain the optimal plane parameters.
5. The method according to claim 4, wherein, the initial plane parameters include an initial normal vector; the obtaining the initial plane parameters of the target spatial plane includes: performing semantic segmentation on at least one of the first image and the second image to obtain a semantic segmentation result; determining the plane type of the target spatial plane according to the semantic segmentation result; determining the initial normal vector according to the plane type.
6. The method according to claim 4, wherein, the initial plane parameters include an initial intercept; the obtaining the initial plane parameters of the target spatial plane includes: Sample multiple candidate initial intercepts within the value range of a preset initial intercept to form multiple candidate initial plane parameters; Based on each of the multiple candidate initial plane parameters, calculate the total position deviation between the projected pixel points and the second edge in the second edge image, and select the candidate initial plane parameter corresponding to the minimum total position deviation as the initial plane parameter.
7. The method according to claim 6, wherein, the sampling within the value range of the initial intercept to obtain multiple candidate initial intercepts includes: Performing uniform sampling within the value range of the initial intercept at a preset interval to obtain multiple candidate initial intercepts.
8. The method according to claim 1 or 2, wherein, the optimization problem is a gradient-based optimization problem, and the partial derivative of the position deviation with respect to the plane parameter is used as the gradient of the optimization problem.
9. A plane estimation device, comprising: An image acquisition module for acquiring a first image and a second image, both of which contain a target space plane, and the shooting perspectives of the first image and the second image are different; A homography transformation matrix determination module for determining the homography transformation matrix of the target space plane between the first image and the second image; wherein, the homography transformation matrix is related to the plane parameters of the target space plane; An edge image extraction module for extracting a first edge image of the target space plane in the first image and a second edge image of the target space plane in the second image; A projected pixel point determination module for determining the projected pixel points corresponding to the pixels of the first edge in the first edge image in the second edge image based on the homography transformation matrix; A position deviation determination module for determining the position deviation between the projected pixel points and the second edge in the second edge image; wherein, the position deviation is a function of the plane parameters of the target space plane; A plane parameter solving module for constructing an optimization problem with minimizing the position deviation as the objective function and solving for the optimal plane parameters.
10. An electronic device, comprising: A processor; and A memory storing a program; wherein, the program includes instructions that, when executed by the processor, cause the processor to execute the plane estimation method according to any one of claims 1-8.
11. A computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the plane estimation method according to any one of claims 1-8.