A plane estimation method, device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure disclose a plane estimation method and device, electronic equipment and storage medium, wherein the method comprises: obtaining video frames of a target video, extracting feature points in each of the video frames, and determining position information of the same feature points in each of the video frames; based on a random sample consensus algorithm, determining a homography matrix of a plane between adjacent video frames according to the position information of each of the feature points in each of the video frames; and determining parameter information of the plane according to the homography matrix of the plane between adjacent video frames. The plane estimation method and device can realize plane estimation of multiple video frames, and the plane estimation effect is good.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to a plane estimation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the prior art, plane estimation in each video frame can be applied to many scenarios, such as 3D reconstruction. When performing plane estimation on multi-frame images, the traditional method is usually as follows: first, obtain 3D point cloud data based on the multi-frame Structure from Motion (SfM) technique; then, perform plane estimation based on the 3D point cloud data.

[0003] The shortcomings of traditional methods include at least the following: the quality of 3D point data depends on the accuracy of SfM, and when the quality of 3D point data is poor, the plane estimation effect is poor. Summary of the Invention

[0004] This disclosure provides a plane estimation method, apparatus, electronic device, and storage medium that can perform plane estimation on multiple video frames with good plane estimation results.

[0005] In a first aspect, embodiments of this disclosure provide a plane estimation method, including:

[0006] Acquire video frames of the target video, extract feature points from each video frame, and determine the position information of the same feature points in each video frame;

[0007] Based on the random sampling consensus algorithm, the homography matrix of the plane between adjacent video frames is determined according to the position information of each feature point in each video frame;

[0008] The parameter information of the plane is determined based on the homography matrix of the plane between adjacent video frames.

[0009] Secondly, embodiments of this disclosure also provide a plane estimation apparatus, comprising:

[0010] The location information determination module is used to acquire video frames of the target video, extract feature points in each video frame, and determine the location information of the same feature points in each video frame.

[0011] The homography matrix determination module is used to determine the homography matrix of the plane between adjacent video frames based on the random sampling consensus algorithm and the position information of each feature point in each video frame.

[0012] The plane parameter determination module is used to determine the parameter information of the plane based on the homography matrix of the plane between adjacent video frames.

[0013] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0014] One or more processors;

[0015] Storage device for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the plane estimation method as described in any embodiment of this disclosure.

[0017] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the plane estimation method as described in any of the embodiments of this disclosure.

[0018] The technical solution of this disclosure involves acquiring video frames of a target video, extracting feature points from each video frame, and determining the position information of the same feature points in each video frame; based on a random sampling consensus algorithm, determining the homography matrix of the plane between adjacent video frames according to the position information of each feature point in adjacent video frames; and determining the parameter information of the plane according to the homography matrix of the plane between adjacent video frames.

[0019] By directly executing the random sampling consensus algorithm across multiple video frames to obtain the parameter information of each plane in each video frame, the errors introduced by the SfM algorithm can be avoided. Furthermore, since multiple video frames contain richer planar information, plane estimation based on video frames can not only quickly and conveniently determine the parameter information of multiple planes, but also achieve better plane estimation results. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1 This is a flowchart illustrating a plane estimation method provided in Embodiment 1 of this disclosure;

[0022] Figure 2 This is a schematic flowchart of a plane estimation method provided in Embodiment 2 of this disclosure;

[0023] Figure 3 This is a schematic flowchart of a plane estimation method provided in Embodiment 3 of this disclosure;

[0024] Figure 4This is a schematic diagram of the structure of a planar estimation device provided in Embodiment 4 of this disclosure;

[0025] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of this disclosure. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0029] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0030] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0031] Example 1

[0032] Figure 1 This is a schematic flowchart illustrating a plane estimation method provided in Embodiment 1 of this disclosure. This embodiment is applicable to situations involving plane estimation of multiple image frames, such as situations involving multiple plane estimations of multiple video frames. The method can be executed by a plane estimation device, which can be implemented in software and / or hardware and can be configured in an electronic device, such as a computer.

[0033] like Figure 1 As shown, the plane estimation method provided in this embodiment includes:

[0034] S110. Obtain video frames of the target video, extract feature points in each video frame, and determine the position information of the same feature points in each video frame.

[0035] The target video can be considered as the video in which the planar information it contains needs to be determined. The target video can be obtained from a preset storage location, or it can be a video captured in real-time.

[0036] The methods for obtaining video frames from the target video may include, but are not limited to: using open-source programs such as ffmpeg to parse each frame of the target video to obtain video frames; or using a pre-written Java program to extract video frames from the target video at predetermined intervals.

[0037] Feature point extraction can be performed directly on the acquired video frames, or the video frames can be filtered first, and then feature point extraction can be performed on the filtered video frames. Video frame filtering can be based on the similarity of the image content contained in the video frames (e.g., for video frames with adjacent content similarity greater than a preset value, only one of them is retained); or based on the interval duration of the video frames (e.g., within a certain time interval, only one video frame is retained); or other methods can be used for video frame filtering. The specific method can be selected according to the application scenario, and will not be exhaustively listed here. By filtering video frames, the efficiency of plane estimation can be guaranteed while ensuring the accuracy of plane estimation.

[0038] The feature points in the video frame may include, but are not limited to, corner points and / or local pixel feature points (such as pixel maximum and minimum points). The specific types of feature points to be extracted can be set according to the application scenario. Different extraction algorithms can be used for different types of feature points. For example, when the feature point is a corner point, the Harris corner detection algorithm can be used for corner point extraction; when the feature point is a local pixel feature point, the Difference of Gaussian (DoG) operator can be used for local pixel feature extraction. This is not an exhaustive list.

[0039] The positional information of feature points within a video frame can be considered as the pixel coordinates of the feature points within the video frame. After determining the feature points in each video frame, the positions of the same feature points in each video frame can be tracked. Furthermore, the positional information of each feature point in each video frame can be determined.

[0040] One approach is to use optical flow algorithms, such as the KLT (Kanade-Lucas-Tomasi Tracking Method), to track the position of the same feature point in each video frame. Alternatively, feature point matching can be performed based on the Oriented Fast and Rotated BRIEF (ORB) feature principle to determine the position information of the same feature point in each video frame. These methods are not exhaustively described here.

[0041] In some alternative approaches, a track can be generated for each feature point to represent its position within each video frame. For example, when there are n video frames, the track corresponding to feature point 1 could be [x...]. 11 ,y 11 ,x 21 ,y 21 ,x 31 ,y 31 ...x n1 ,y n1 ]; where x represents the horizontal coordinate of a pixel; y represents the vertical coordinate of a pixel; in the subscripts of x and y, the first number can represent the frame number of the video frame, and the second number can represent the feature point label, for example x 21 It can represent the pixel x-coordinate of feature point 1 in the second video frame.

[0042] S120. Based on the random sampling consensus algorithm, determine the homography matrix of the plane between adjacent video frames according to the position information of each feature point in each video frame.

[0043] The random sampling consensus algorithm can be considered as an algorithm that calculates mathematical model parameters that conform to normal data based on a sample dataset containing normal and abnormal data. In this embodiment of the disclosure, for the case of determining the homography matrix of a certain plane between adjacent video frames, the track of feature points can be used as the sample dataset; the track of feature points belonging to the plane can be used as normal data, and the track of feature points not belonging to the plane can be used as abnormal data; the homography matrix of the plane between adjacent video frames can be used as the calculated mathematical model parameters that conform to normal data.

[0044] For example, when there are n video frames, the mathematical model parameters corresponding to plane 1 may include the homography matrix of the plane between the 1st and 2nd video frames, the homography matrix of the plane between the 2nd and 3rd video frames, ..., the homography matrix of the plane between the (n-1)th and nth video frames. These n-1 homography matrices can be used as the mathematical model parameters of plane 1 that conform to normal data.

[0045] The homography matrix of a plane can be considered as a perspective transformation matrix, used to represent the perspective transformation of a plane from one view to another. The homography matrix can define intrinsic parameters (e.g., camera focal length, lens distortion) and extrinsic parameters (e.g., rotation matrix, translation matrix) of the camera used to capture the video. The parameters in the homography matrix can be calculated based on the positional information of identical feature points between two frames. It can be considered that, based on the positional information of feature points contained in the track within each video frame, the homography matrix of the plane between every two adjacent video frames can be calculated.

[0046] In this embodiment, the position information of each feature point in each video frame can be used as input to the random sampling consensus algorithm. The algorithm randomly samples feature points to determine those belonging to a certain plane, and based on the track of the determined feature points belonging to that plane, outputs the homography matrix of the plane between adjacent video frames. By directly performing random sampling consensus on the entire video to obtain the homography matrix of the plane, a foundation can be laid for determining the parameter information of the plane.

[0047] S130. Determine the parameter information of the plane based on the homography matrix of the plane between adjacent video frames.

[0048] In this context, the parameter information of a plane can be considered as the parameters in the expression of the plane in the spatial coordinate system. For example, if the expression of plane 1 in the spatial coordinate system is Ax + By + Cz + D = 0, then A, B, C, and D can be considered as the parameter information of the plane.

[0049] In this embodiment of the disclosure, the homography matrix of the plane between adjacent video frames can be decomposed to obtain the camera intrinsic and extrinsic parameters defined therein. Furthermore, based on the plane's position information in the video frame, and the camera's intrinsic and extrinsic parameters, the parameters of the plane's expression in the spatial coordinate system can be obtained, thus acquiring the plane's parameter information.

[0050] The homography matrix of the inter-frame plane can be decomposed in various ways, such as: the method of Faugeras (1988) in ORB-SLAM2; or the method of decomposed Homography Mat in OpenCV, where the decompose Homography Mat function can be implemented using the method of INRIA (2007), which will not be exhaustively listed here.

[0051] By directly executing the random sampling consensus algorithm across multiple video frames to obtain the parameter information of each plane in each video frame, the errors introduced by the SfM algorithm can be avoided. Furthermore, since multiple video frames contain richer planar information, plane estimation based on video frames can not only quickly and conveniently determine the parameter information of multiple planes, but also achieve better plane estimation results.

[0052] In some optional implementations, after determining the parameter information of the plane, the method further includes: determining the position and orientation information of the virtual object based on the parameter information of the plane; and displaying the virtual object and the plane in a preset video frame based on the position and orientation information.

[0053] The virtual object can be, for example, the appearance of a virtual artificial intelligence (AI), or an interactive virtual control. The position and posture information of the virtual object can include, but is not limited to, its position and rotation angle in a spatial coordinate system. The preset video frames can be, for example, video frames containing the plane to be displayed in association, or video frames within a preset time period. The association display methods can include, but are not limited to, displaying the virtual object below the plane, or hovering it above the plane, etc., with related positions.

[0054] After determining the plane's parameters, the position and rotation angle of the virtual object in the spatial coordinate system can be determined based on the plane's expression in the spatial coordinate system and the way the virtual object and the plane are displayed together. For example, if the virtual object is displayed on the plane, it can be vertically displaced by a certain value in the positive direction based on its corresponding position and rotation angle on the plane in the spatial coordinate system.

[0055] After determining the position and pose information of the virtual object, the pixel position and rotation angle of the virtual object in the preset video frame can be determined based on its position and rotation angle in the spatial coordinate system, as well as the transformation relationship between the plane in the spatial coordinate system and the pixel coordinate system in the preset video frame. Furthermore, the virtual object can be rendered based on this pixel position and rotation angle information. Since the position and pose information of the virtual object in the spatial coordinate system is determined according to the association display method between the virtual object and the plane, rendering the virtual object allows for the association display of the virtual object and the plane within the preset video frame.

[0056] Among these alternative implementations, virtual objects can be rendered based on planes contained in a given video, thereby enabling augmented reality on video frames to improve the user experience.

[0057] The technical solution of this disclosure involves acquiring video frames of a target video, extracting feature points from each video frame, and determining the position information of the same feature points in each video frame; based on a random sampling consensus algorithm, determining the homography matrix of the plane between adjacent video frames according to the position information of each feature point in adjacent video frames; and determining the parameter information of the plane according to the homography matrix of the plane between adjacent video frames.

[0058] By directly executing the random sampling consensus algorithm across multiple video frames to obtain the parameter information of each plane in each video frame, the errors introduced by the SfM algorithm can be avoided. Furthermore, since multiple video frames contain richer planar information, plane estimation based on video frames can not only quickly and conveniently determine the parameter information of multiple planes, but also achieve better plane estimation results.

[0059] Example 2

[0060] This embodiment can be combined with various optional schemes in the plane estimation method provided in the above embodiments. The plane estimation method provided in this embodiment details the steps for determining the homography matrix of planes between adjacent video frames. By iteratively extracting a preset number of first feature points to estimate the initial homography matrix, and determining the number of inliers in the second feature points based on the initial homography matrix, the case with the maximum number of inliers can be iteratively identified. Furthermore, based on the position information of each inlier in the case with the maximum number of inliers, the homography matrix of the plane to which these inliers belong can be determined. This enables direct random sampling consistency of the video to quickly and conveniently determine the parameter information of multiple planes in multiple frames of images.

[0061] Figure 2 This is a schematic flowchart of a plane estimation method provided in Embodiment 2 of this disclosure. Figure 2 As shown, the plane estimation method provided in this embodiment includes:

[0062] S210. Obtain video frames of the target video, extract feature points in each video frame, and determine the position information of the same feature points in each video frame.

[0063] S220. A preset number of feature points are extracted cyclically as the first feature points, and the unextracted feature points are used as the second feature points.

[0064] The process of determining the homography matrix of the plane between adjacent video frames based on the position information of each feature point in each video frame, according to random sampling consistency, may include steps S220-S250. Since the homography matrix has 8 degrees of freedom, at least 4 preset parameters are required to determine the 8 unknown parameters in the homography matrix. It can be assumed that a preset number of first feature points can be randomly selected in each iteration.

[0065] S230. Based on the position information of the first feature point in adjacent video frames, determine the initial homography matrix of the plane between adjacent video frames.

[0066] Specifically, the position information of a preset number of first feature points in every two adjacent video frames can be determined based on the track of the preset number of first feature points. Furthermore, the initial homography matrix of the plane between adjacent video frames can be determined based on the position information of the preset number of first feature points in every two adjacent video frames.

[0067] S240. Based on the position information of the second feature point in adjacent video frames and the initial homography matrix, determine whether the second feature point matches each initial homography matrix.

[0068] After determining a set of initial homography matrices, each second feature point can be used to verify whether the initial homography matrix is ​​optimal. Specifically, if the positional information of any two adjacent video frames in the track of a certain second feature point conforms to the mapping relationship represented by the initial homography matrix, then the second feature point can be considered to match each initial homography matrix in the set. The more second feature points that match each initial homography matrix, the better the initial homography matrix is ​​considered to be, and the more accurately it can represent the projection relationship of a plane between video frames.

[0069] In some optional implementations, based on the position information of the second feature point in adjacent video frames and the initial homography matrix, it is determined whether the second feature point matches each initial homography matrix. This includes: determining the reprojection position information of the second feature point in the adjacent video frames based on the position information of the second feature point in the earlier-ordered video frames and the initial homography matrix of the plane between adjacent video frames; determining the reprojection error based on the position information of the second feature point in the adjacent video frames and the reprojection position information; and determining whether the second feature point matches each initial homography matrix based on the reprojection error of the second feature point in every two adjacent video frames.

[0070] First, the position information of the second feature point in the earlier-ordered video frame among adjacent video frames can be substituted into the initial homography matrix of the inter-frame plane to obtain the reprojection position information of the second feature point in the later-ordered video frame among adjacent video frames. Second, the reprojection error can be determined based on the pixel distance (e.g., Euclidean distance of pixel coordinates) between the position information of the second feature point in the later-ordered video frame and the reprojection position information. Third, the final error value can be determined by calculating the average reprojection error, determining the median reprojection error, or selecting the maximum / minimum error among the reprojection errors. Finally, based on the final error value, it can be determined whether the second feature point matches any of the initial homography matrices. For example, if the final error value is less than a certain set value, then the second feature point is considered to match any of the initial homography matrices.

[0071] For example, assuming there are n video frames, the initial homography matrix of the plane between the 1st and 2nd video frames is H1, ..., and the initial homography matrix of the plane between the (n-1)th and nth video frames is H... n-1 A total of n-1 initial homography matrices are determined. Feature point 1 belongs to the second feature point, and the track corresponding to feature point 1 can be [x]. 11 ,y 11 ,x 21 ,y 21 ,x 31 ,y 31 ...x n1 ,y n1 ].

[0072] Then, determine whether feature point 1 matches the initial homography matrix H1-H. n-1 Matching can include: for H1, matching the position information x of feature point 1 in the first frame. 11 ,y 11 Substituting H1, we obtain the reprojection position information x in the second frame. 21 ',y 21 ';Calculate x 21 ,y 21 and x 21 ',y 21 The reprojection error w1 between the two points (e.g., the Euclidean distance between the two points); based on the above steps, adaptively calculate H2-H. n-1 The corresponding reprojection error w2-w n-1 According to w1-w n-1 Determine whether feature point 1 corresponds to the initial homography matrix H1-H n-1 match.

[0073] The process of determining whether a second feature point matches any initial homography matrix based on the reprojection error of the second feature point in every two adjacent video frames includes: determining the average reprojection error based on the reprojection error of the second feature point in every two adjacent video frames; and determining whether the second feature point matches any initial homography matrix based on the average reprojection error and a preset threshold. Specifically, for example, if the average reprojection error is less than the preset threshold, the second feature point is considered to match any initial homography matrix. The preset threshold can be set based on experimental or empirical values.

[0074] Among these optional implementations, by determining each reprojection error based on the track of the second feature point and each initial homography matrix, it is possible to determine whether the second feature point matches each initial homography matrix based on each reprojection error.

[0075] S250. If so, the second feature point is taken as an interior point, and the cyclic extraction stops when the number of cycles reaches the preset number, and the target cycle with the most interior points in each cycle is determined.

[0076] In this embodiment, the randomly selected preset number of first feature points can be feature points belonging to the same plane or feature points not belonging to the same plane. Generally, when the first feature points belong to the same plane, each second feature point belonging to that plane can be matched with each initial homography matrix; when the first feature points do not belong to the same plane, then basically only a few second feature points can be matched with each initial homography matrix.

[0077] The number of iterations can be preset based on experience or experimental values. By performing a preset number of iterations, the first feature point can be extracted, the initial homography matrices of the plane can be determined, and the number of second feature points as interior points can be determined in each iteration. Based on the position information of each interior point in adjacent video frames in the iteration with the largest number of interior points, the initial homography matrices can be determined, thus obtaining the optimal solution for the initial homography matrix.

[0078] S260. Based on the position information of the interior points determined in the target loop in adjacent video frames, determine the homography matrix of the plane between adjacent video frames.

[0079] In this embodiment, after determining the optimal solution of each initial homography matrix, the tracks of all interior points can be combined to optimize each initial homography matrix, thereby obtaining the final homography matrix of the adjacent video frame interplane.

[0080] S270. Determine the parameter information of the plane based on the homography matrix of the plane between adjacent video frames.

[0081] The technical solution of this disclosure describes in detail the steps for determining the homography matrix of the plane between adjacent video frames. By iteratively extracting a preset number of first feature points to estimate the initial homography matrix, and determining the number of inliers among the second feature points based on the initial homography matrix, the case with the maximum number of inliers can be iteratively identified. Furthermore, based on the position information of each inlier in the case with the maximum number of inliers, the homography matrix of the plane to which these inliers belong can be determined. This enables direct random sampling consistency of the video to quickly and conveniently determine the plane parameter information in multiple frames of images.

[0082] Furthermore, the plane estimation method provided in this embodiment belongs to the same concept as the plane estimation method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.

[0083] Example 3

[0084] This embodiment can be combined with various optional schemes in the plane estimation method provided in the above embodiments. The plane estimation method provided in this embodiment describes the steps for determining the homography matrix of multiple planes in a video. By determining the parameter information of any plane in the video, the feature points used to determine that plane can be removed, and random sampling consistency can be performed cyclically using the remaining feature points, enabling the estimation of multiple planes appearing in the video. This allows for the rapid and convenient determination of the parameter information of multiple planes in the video.

[0085] Figure 3 This is a schematic flowchart of a plane estimation method provided in Embodiment 3 of this disclosure. Figure 3 As shown, the plane estimation method provided in this embodiment includes:

[0086] S310. Obtain video frames of the target video, extract feature points in each video frame, and determine the position information of the same feature points in each video frame.

[0087] S320. Based on the random sampling consensus algorithm, determine the homography matrix of the current plane between adjacent video frames according to the position information of each feature point in each video frame.

[0088] Specifically, the homography matrix of a currently determined plane between adjacent video frames can be used as the homography matrix of the current plane between adjacent video frames. The method for determining the homography matrix of the current plane between adjacent video frames can be the same as the method for determining the homography matrix of any plane between adjacent video frames, as described above, and will not be repeated here.

[0089] S330. Loop through the feature points and remove the feature points used to determine the homography matrix of the current plane to obtain the remaining feature points.

[0090] In this embodiment, since the video can contain planar information of multiple planes, after determining the homography matrix of the current plane between adjacent video frames, the feature points used to determine the homography matrix of the current plane can be removed, and the random sampling consensus algorithm can be continued to determine the parameter information of other planes.

[0091] S340. Based on the random sampling consensus algorithm, determine the homography matrix of the new plane in the adjacent video frame according to the position information of each remaining feature point in the adjacent video frame, and stop the cyclic removal when the number of remaining feature points is less than the preset number.

[0092] In this embodiment, the preset number can be set based on experimental or empirical values. When the number of remaining feature points is less than the preset number, it can be assumed that all evaluable planes in the video have had their parameter information evaluated. At this point, the removal of feature points used to determine the homography matrix of the current plane can be stopped, and the random sampling consensus algorithm can be stopped.

[0093] S350. Determine the parameter information of the plane based on the homography matrix of the plane between adjacent video frames.

[0094] In this embodiment, the parameter information of a plane can be determined when determining the homography matrix of each plane between adjacent video frames; or the parameter information of each plane can be determined when determining the parameter information of all planes.

[0095] The technical solution of this disclosure describes the steps for determining the homography matrix of multiple planes in a video. After determining the parameter information of any plane in the video, the feature points used to determine that plane can be removed, and random sampling consistency can be performed cyclically using the remaining feature points, enabling the estimation of multiple planes appearing in the video. This allows for the rapid and convenient determination of the parameter information of multiple planes in the video.

[0096] Furthermore, the plane estimation method provided in this embodiment belongs to the same concept as the plane estimation method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.

[0097] Example 4

[0098] Figure 4This is a schematic diagram of a plane estimation device provided in Embodiment 4 of this disclosure. The plane estimation device provided in this embodiment is suitable for performing plane estimation on multiple frames of images, such as performing multiple plane estimations on a video.

[0099] like Figure 4 As shown, the plane estimation device provided in this embodiment includes:

[0100] The location information determination module 410 is used to acquire video frames of the target video, extract feature points in each video frame, and determine the location information of the same feature points in each video frame.

[0101] The homography matrix determination module 420 is used to determine the homography matrix of the plane between adjacent video frames based on the random sampling consensus algorithm and the position information of each feature point in each video frame.

[0102] The plane parameter determination module 430 is used to determine the plane parameter information based on the homography matrix of the planes between adjacent video frames.

[0103] In some alternative implementations, the homography matrix determination module includes:

[0104] The extraction unit is used to cyclically extract a preset number of feature points as first feature points and use the unextracted feature points as second feature points.

[0105] The initial matrix determination unit is used to determine the initial homography matrix of the plane between adjacent video frames based on the position information of the first feature point in adjacent video frames;

[0106] The judgment unit is used to determine whether the second feature point matches each initial homography matrix based on the position information of the second feature point in adjacent video frames and the initial homography matrix.

[0107] The inlier determination unit is used to determine the inlier if the second feature point matches each initial homography matrix, until the number of iterations reaches a preset number and the cyclic extraction stops, and the target cycle with the most inliers in each iteration is determined.

[0108] The final matrix determination unit is used to determine the homography matrix of the plane between adjacent video frames based on the position information of the inliers determined in the target loop in adjacent video frames.

[0109] In some alternative implementations, the decision unit includes:

[0110] The reprojection subunit is used to determine the reprojection position information of the second feature point in the next video frame based on the position information of the second feature point in the video frame with the earlier sorting order and the initial homography matrix of the plane between adjacent video frames.

[0111] The error determination subunit is used to determine the reprojection error based on the position information of the second feature point in the later video frame in the adjacent video frames and the reprojection position information.

[0112] The judgment sub-unit is used to determine whether the second feature point matches each initial homography matrix based on the reprojection error of the second feature point in every two adjacent video frames.

[0113] In some optional implementations, the sub-unit is determined specifically for:

[0114] The average reprojection error is determined based on the reprojection error of the second feature point in every two adjacent video frames; based on the average reprojection error and a preset threshold, it is determined whether the second feature point matches each initial homography matrix.

[0115] In some alternative implementations, the homography matrix determination module includes:

[0116] The removal unit is used to cyclically remove the feature points used to determine the homography matrix of the current plane after determining the homography matrix of the current plane between adjacent video frames, so as to obtain the remaining feature points.

[0117] The matrix determination unit is used to determine the homography matrix of a new plane in adjacent video frames based on the position information of each remaining feature point in adjacent video frames, according to the random sampling consensus algorithm, until the number of remaining feature points is less than a preset number and the cyclic removal stops.

[0118] In some alternative implementations, the plane estimation device further includes:

[0119] The virtual display module is used to determine the position and orientation information of the virtual object based on the parameter information of the plane; and to associate and display the virtual object with the plane in preset video frames based on the position and orientation information.

[0120] The plane estimation apparatus provided in this disclosure can execute the plane estimation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method execution.

[0121] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0122] Example 5

[0123] The following is for reference. Figure 5 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 5 The diagram below shows the structure of the terminal device or server 500. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0124] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 506 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0125] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0126] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 509, or installed from storage device 506, or installed from ROM 502. When the computer program is executed by processing device 501, it performs the functions defined above in the plane estimation method of embodiments of this disclosure.

[0127] The electronic device provided in this embodiment and the plane estimation method provided in the above embodiments belong to the same disclosed concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0128] Example 6

[0129] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the plane estimation method provided in the above embodiments.

[0130] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory (FLASH), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0131] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0132] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0133] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to:

[0134] The process involves acquiring video frames of the target video, extracting feature points from each video frame, and determining the positional information of identical feature points within each video frame. Based on a random sampling consensus algorithm, the homography matrix of the plane between adjacent video frames is determined according to the positional information of each feature point in each video frame. Finally, the parameter information of the plane is determined based on the homography matrix of the plane between adjacent video frames.

[0135] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0137] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units and modules do not, in certain circumstances, constitute a limitation on the unit or module itself.

[0138] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0139] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0140] According to one or more embodiments of this disclosure, [Example 1] provides a plane estimation method, the method comprising:

[0141] Acquire video frames of the target video, extract feature points from each video frame, and determine the position information of the same feature points in each video frame;

[0142] Based on the random sampling consensus algorithm, the homography matrix of the plane between adjacent video frames is determined according to the position information of each feature point in each video frame;

[0143] The parameter information of the plane is determined based on the homography matrix of the plane between adjacent video frames.

[0144] According to one or more embodiments of this disclosure, [Example 2] provides a plane estimation method, which further includes:

[0145] In some optional implementations, the step of determining the homography matrix of the plane between adjacent video frames based on the random sampling consensus algorithm and the position information of each feature point in each video frame includes:

[0146] A preset number of feature points are extracted cyclically as the first feature points, and the unextracted feature points are used as the second feature points.

[0147] Based on the position information of the first feature point in adjacent video frames, the initial homography matrix of the plane between adjacent video frames is determined;

[0148] Based on the position information of the second feature point in adjacent video frames and the initial homography matrix, determine whether the second feature point matches each of the initial homography matrices;

[0149] If so, the second feature point is taken as an interior point, and the cyclic extraction stops when the number of iterations reaches the preset number, and the target loop with the largest number of interior points in each iteration is determined.

[0150] Based on the position information of the inliers determined in the target loop in adjacent video frames, the homography matrix of the plane between adjacent video frames is determined.

[0151] According to one or more embodiments of this disclosure, [Example 3] provides a plane estimation method, further comprising:

[0152] In some optional implementations, determining whether the second feature point matches each of the initial homography matrices based on the position information of the second feature point in adjacent video frames and the initial homography matrix includes:

[0153] Based on the position information of the second feature point in the preceding video frame among adjacent video frames, and the initial homography matrix of the plane between adjacent video frames, the reprojection position information of the second feature point in the following video frame among adjacent video frames is determined.

[0154] The reprojection error is determined based on the position information of the second feature point in the later video frame in the adjacent video frames and the reprojection position information.

[0155] Based on the reprojection error of the second feature point in every two adjacent video frames, it is determined whether the second feature point matches each of the initial homography matrices.

[0156] According to one or more embodiments of this disclosure, [Example 4] provides a plane estimation method, further comprising:

[0157] In some optional implementations, determining whether the second feature point matches each of the initial homography matrices based on the reprojection error of the second feature point in every two adjacent video frames includes:

[0158] The average reprojection error is determined based on the reprojection error of the second feature point in every two adjacent video frames.

[0159] Based on the average reprojection error and the preset threshold, it is determined whether the second feature point matches each of the initial homography matrices.

[0160] According to one or more embodiments of this disclosure, Example 5 provides a plane estimation method, which further includes:

[0161] In some optional implementations, determining the homography matrix of adjacent video frame planes includes:

[0162] After determining the homography matrix of the current plane between adjacent video frames, the feature points used to determine the homography matrix of the current plane are removed from the feature points in a loop to obtain the remaining feature points;

[0163] Based on the random sampling consensus algorithm, the homography matrix of the new plane in the adjacent video frames is determined according to the position information of each remaining feature point in the adjacent video frames, and the cyclic removal stops when the number of remaining feature points is less than a preset number.

[0164] According to one or more embodiments of this disclosure, Example Six provides a plane estimation method, further comprising:

[0165] In some alternative implementations, after determining the parameter information of the plane, the following steps are also included:

[0166] Based on the parameter information of the plane, determine the position and orientation information of the virtual object;

[0167] Based on the position and pose information, the virtual object is associated with and displayed on the plane in a preset video frame.

[0168] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0169] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0170] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A plane estimation method, characterized in that, include: The target video is acquired in multiple frames, feature points in each of the multiple video frames are extracted, and the position information of the same feature points in each video frame is determined. Based on the random sampling consensus algorithm, the homography matrix of the plane between every two adjacent video frames is determined according to the position information of each feature point in each video frame. The homography matrix of the plane is the perspective transformation matrix of the plane, which is used to characterize the perspective transformation of the plane in different views. Based on the homography matrix of multiple adjacent video frames in the target video, the parameter information of the plane is determined. The parameter information of the plane is a set of parameters used to characterize the position and orientation of the plane in the spatial coordinate system. Based on the parameter information of the plane, the position and orientation information of the virtual object is determined, including the position and rotation angle of the virtual object in the spatial coordinate system. Based on the position and pose information, the virtual object is associated with and displayed on the plane in a preset video frame.

2. The method according to claim 1, characterized in that, The method based on random sampling consensus algorithm determines the homography matrix of the plane between adjacent video frames according to the position information of each feature point in each video frame, including: A preset number of feature points are extracted cyclically as the first feature points, and the unextracted feature points are used as the second feature points. Based on the position information of the first feature point in adjacent video frames, the initial homography matrix of the plane between adjacent video frames is determined; Based on the position information of the second feature point in adjacent video frames and the initial homography matrix, determine whether the second feature point matches each of the initial homography matrices; If so, the second feature point is taken as an interior point, and the cyclic extraction stops when the number of iterations reaches the preset number, and the target loop with the largest number of interior points in each iteration is determined. Based on the position information of the inliers determined in the target loop in adjacent video frames, the homography matrix of the plane between adjacent video frames is determined.

3. The method according to claim 2, characterized in that, The step of determining whether the second feature point matches each of the initial homography matrices based on the position information of the second feature point in adjacent video frames and the initial homography matrix includes: Based on the position information of the second feature point in the preceding video frame among adjacent video frames, and the initial homography matrix of the plane between adjacent video frames, the reprojection position information of the second feature point in the following video frame among adjacent video frames is determined. The reprojection error is determined based on the position information of the second feature point in the later video frame in the adjacent video frames and the reprojection position information. Based on the reprojection error of the second feature point in every two adjacent video frames, it is determined whether the second feature point matches each of the initial homography matrices.

4. The method according to claim 3, characterized in that, The step of determining whether the second feature point matches each of the initial homography matrices based on the reprojection error of the second feature point in every two adjacent video frames includes: The average reprojection error is determined based on the reprojection error of the second feature point in every two adjacent video frames. Based on the average reprojection error and the preset threshold, it is determined whether the second feature point matches each of the initial homography matrices.

5. The method according to claim 1, characterized in that, The determination of the homography matrix of adjacent video frame planes includes: After determining the homography matrix of the current plane between adjacent video frames, the feature points used to determine the homography matrix of the current plane are removed from the feature points in a loop to obtain the remaining feature points; Based on the random sampling consensus algorithm, the homography matrix of the new plane in the adjacent video frames is determined according to the position information of each remaining feature point in the adjacent video frames, and the cyclic removal stops when the number of remaining feature points is less than a preset number.

6. A plane estimation device, characterized in that, include: The location information determination module is used to acquire multiple video frames of the target video, extract feature points in each of the multiple video frames, and determine the location information of the same feature points in each of the video frames. The homography matrix determination module is used to determine the homography matrix of the plane between every two adjacent video frames in the multi-frame video frame based on the random sampling consensus algorithm and the position information of each feature point in each video frame. The homography matrix of the plane is the perspective transformation matrix of the plane, which is used to characterize the perspective transformation of the plane in different views. The plane parameter determination module is used to determine the parameter information of the plane based on the homography matrix of multiple sets of adjacent video frames in the target video. The parameter information of the plane is a set of parameters used to characterize the position and orientation of the plane in the spatial coordinate system. The virtual display module is used to determine the position and orientation information of the virtual object based on the parameter information of the plane. The position and orientation information of the virtual object includes the position and rotation angle of the virtual object in the spatial coordinate system. Based on the position and pose information, the virtual object is associated with and displayed on the plane in a preset video frame.

7. The apparatus according to claim 6, characterized in that, The homography matrix determination module includes: The extraction unit is used to cyclically extract a preset number of feature points as first feature points and use the unextracted feature points as second feature points. The initial matrix determination unit is used to determine the initial homography matrix of the plane between adjacent video frames based on the position information of the first feature point in adjacent video frames; The judgment unit is used to determine whether the second feature point matches each of the initial homography matrices based on the position information of the second feature point in adjacent video frames and the initial homography matrix. An interior point determination unit is used to determine if the second feature point matches each of the initial homography matrices, and to stop the cyclic extraction when the number of interior points is the maximum. The final matrix determination unit is used to determine the homography matrix of the plane between adjacent video frames based on the position information of the inliers determined when the loop stops.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the plane estimation method as described in any one of claims 1-5.

9. A storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to perform the plane estimation method as described in any one of claims 1-5.

Images