A camera pose estimation method, apparatus, device, medium and product

Abstract

The application discloses a camera pose estimation method, device, equipment, medium and product, relates to the technical field of pose estimation, and comprises the following steps: extracting a matching parallel straight line pair and a matching conic curve pair between a first image and a second image; taking a rotation quaternion between the first image and the second image as a parameter, constructing a direction vector conversion error function according to each matching parallel straight line pair and a camera intrinsic parameter matrix, and solving the function value minimum as a target to obtain an intermediate rotation quaternion; taking the rotation quaternion between the first image and the second image as the parameter, constructing a conic curve projection error function according to each matching conic curve pair, and fusing the direction vector conversion error function and the conic curve projection error function to obtain a joint objective function; constructing a Lagrange optimization function according to the joint objective function and a Lagrange multiplier; and performing extreme value solving on the Lagrange optimization function according to the intermediate rotation quaternion, so that the pose of the camera is determined, and the pose precision is improved.

Description

Technical Field

[0001] This invention relates to the field of pose estimation technology, and in particular to a method, apparatus, device, medium and product for gauging the pose of a camera. Background Technology

[0002] Camera pose estimation is a core fundamental problem in computer vision, aiming to determine the camera's position and orientation in three-dimensional space from a two-dimensional image. It is a crucial support for tasks such as visual SLAM (Simultaneous Localization and Mapping), 3D reconstruction, augmented reality, and robot localization.

[0003] Existing technologies typically solve camera pose by detecting feature points and matching them. This method relies on the accuracy of feature point detection and the quality of matching. In complex environments such as varying lighting and sparse textures, the accuracy of camera pose estimation results is poor. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and product for camera pose estimation, in order to improve the accuracy of camera pose estimation results.

[0005] In a first aspect, embodiments of the present invention provide a camera pose estimation method, comprising:

[0006] Acquire a first image and a second image captured by the camera, as well as a camera intrinsic parameter matrix; the second image was acquired before the first image.

[0007] Extract matching pairs of parallel lines and matching pairs of quadratic curves between the first image and the second image; the matching pairs of parallel lines include a pair of parallel lines in the first image and a pair of parallel lines in the second image; the matching pairs of quadratic curves include a quadratic curve in the first image and a quadratic curve in the second image.

[0008] Using the rotation quaternion between the first image and the second image as a parameter, a direction vector transformation error function is constructed based on each of the matching parallel line pairs and the camera intrinsic parameter matrix. The intermediate rotation quaternion is obtained by solving the problem with the goal of minimizing the function value of the direction vector transformation error function.

[0009] If a pair of matching quadratic curves exists, then using the rotation quaternion between the first image and the second image as a parameter, a quadratic curve projection error function is constructed based on each pair of matching quadratic curves and the camera intrinsic parameter matrix. The direction vector transformation error function and the quadratic curve projection error function are then fused to obtain a joint objective function.

[0010] Using the unit module as a constraint, a Lagrange optimization function is constructed based on the joint objective function and the Lagrange multipliers. The intermediate rotated quaternion is used as the initial value for iteration. The extreme value of the Lagrange optimization function is then solved to obtain the target rotated quaternion.

[0011] The camera pose is determined based on the target rotation quaternion.

[0012] Secondly, embodiments of the present invention also provide a camera pose estimation device, comprising:

[0013] The acquisition module is used to acquire a first image and a second image captured by the camera, as well as a camera intrinsic parameter matrix; the acquisition time of the second image is before that of the first image.

[0014] An extraction module is used to extract at least one pair of matching parallel lines and at least one pair of matching quadratic curves between the first image and the second image; the pair of matching parallel lines includes a pair of parallel lines in the first image and a pair of parallel lines in the second image; the pair of matching quadratic curves includes a quadratic curve in the first image and a quadratic curve in the second image.

[0015] The first solution module is used to construct a direction vector transformation error function based on the rotation quaternion between the first image and the second image, according to each of the matching parallel line pairs and the camera intrinsic parameter matrix, and to solve the problem with the goal of minimizing the function value of the direction vector transformation error function to obtain the intermediate rotation quaternion.

[0016] The fusion module is used to construct a quadratic curve projection error function based on the rotation quaternion between the first image and the second image, using the rotation quaternion as a parameter, according to each of the matching quadratic curve pairs and the camera intrinsic parameter matrix, and to fuse the direction vector transformation error function and the quadratic curve projection error function to obtain a joint objective function if a matching quadratic curve pair exists.

[0017] The second solution module is used to take the unit module as a constraint condition for the modulus of the rotated quaternion, construct a Lagrange optimization function based on the joint objective function and the Lagrange multipliers, and use the intermediate rotated quaternion as the initial value for iteration to solve the extreme value of the Lagrange optimization function to obtain the target rotated quaternion.

[0018] The pose determination module is used to determine the pose of the camera based on the target rotation quaternion.

[0019] Thirdly, embodiments of the present invention also provide an electronic device, comprising:

[0020] At least one processor; and

[0021] A memory that is communicatively connected to at least one processor; wherein

[0022] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the camera pose estimation method provided in any embodiment of the present invention.

[0023] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions for causing a processor to execute and implement the camera pose estimation method of any embodiment of the present invention.

[0024] Fifthly, embodiments of the present invention also provide a computer program product, characterized in that the computer program product includes a computer program, which, when executed by a processor, implements the camera pose estimation method of any embodiment of the present invention.

[0025] This invention constructs a direction vector transformation error function by matching parallel line pairs and solving the intermediate rotation quaternion to obtain a reliable initial solution for the rotation quaternion. In the case of a quadratic curve, the quadratic curve projection error is fused to form a joint objective function, and then the Lagrange multiplier is used to incorporate the unit modulus constraint for extreme value solving, thereby achieving complementarity between the two types of geometric constraints and improving the accuracy and robustness of camera pose estimation, especially in special scenarios such as sparse texture and varying illumination.

[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a camera pose estimation method according to Embodiment 1 of the present invention;

[0029] Figure 2 This is a flowchart of a camera pose estimation method according to Embodiment 2 of the present invention;

[0030] Figure 3 This is a schematic diagram of the structure of a camera pose estimation device according to Embodiment 3 of the present invention;

[0031] Figure 4 This is a structural diagram of an electronic device that implements a camera pose estimation method according to an embodiment of the present invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first" and "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] In the technical solutions of the embodiments of the present invention, the acquisition, storage and application of the first image and the second image, etc., all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0035] Example 1

[0036] Figure 1 This is a flowchart of a camera pose estimation method provided in Embodiment 1 of the present invention. This embodiment can be applied to the case of estimating the pose of a camera. The method can be executed by a camera pose estimation device, which can be implemented in hardware and / or software and specifically configured in an electronic device.

[0037] See Figure 1 The camera pose estimation method shown includes:

[0038] S101. Acquire the first and second images captured by the camera, as well as the camera intrinsic parameter matrix.

[0039] S102. Extract the matching pairs of parallel lines and matching pairs of quadratic curves between the first image and the second image.

[0040] S103. Using the rotation quaternion between the first image and the second image as a parameter, construct a direction vector transformation error function based on each of the matching parallel line pairs and the camera intrinsic parameter matrix, and solve the problem with the goal of minimizing the function value of the direction vector transformation error function to obtain the intermediate rotation quaternion.

[0041] S104. If a pair of matching quadratic curves exists, then using the rotation quaternion between the first image and the second image as a parameter, a quadratic curve projection error function is constructed based on each pair of matching quadratic curves and the camera intrinsic parameter matrix. The direction vector transformation error function and the quadratic curve projection error function are then fused to obtain a joint objective function.

[0042] S105. Using the unit module as a constraint condition, construct a Lagrange optimization function based on the joint objective function and the Lagrange multipliers, and use the intermediate rotated quaternion as the initial value for iteration to solve the extreme value of the Lagrange optimization function to obtain the target rotated quaternion.

[0043] S106. Determine the pose of the camera based on the target rotation quaternion.

[0044] In this embodiment, the first image and the second image can be two frames that are consecutive in time; wherein, the second image is acquired before the first image. The matching parallel line pairs include a pair of parallel lines in the first image and a pair of parallel lines in the second image; the parallel line pairs include two parallel line segments in an image; the matching quadratic curve pairs include a quadratic curve in the first image and a quadratic curve in the second image.

[0045] The direction vector transformation error function characterizes the error between the direction vector of the second image and the direction vector of the first image after rotation using a rotation quaternion; the intermediate rotation quaternion can be the result obtained by solving for the direction vector transformation error. The quadratic curve projection error function represents the error between the quadratic curve in the second image and the quadratic curve in the first image after rotation using a rotation quaternion; the joint objective function is the fusion result of the direction vector transformation error function and the quadratic curve projection error function; the target rotation quaternion can be the finally determined quaternion. The constraint that the rotation quaternion has a unit modulus can be that the modulus of the rotation quaternion is constrained to be a unit modulus of 1.

[0046] Specifically, the process involves acquiring a first and second image captured by the camera, along with the camera's intrinsic parameter matrix. A specific algorithm is used to extract matching parallel line pairs and matching quadratic curve pairs between the first and second images. Using the rotation quaternion between the first and second images as parameters, a direction vector transformation error function is constructed based on each matching parallel line pair and the camera intrinsic parameter matrix. The Levenberg-Marquardt (LM) algorithm is used to solve this function, aiming to minimize its value, to obtain the intermediate rotation quaternion. If matching quadratic curve pairs exist, then each... The curve parameters of each quadratic curve in the matched quadratic curve pair are transformed into a corresponding quadratic form matrix; the curve parameters may include, but are not limited to, center point coordinates, major semi-axis, minor semi-axis, and rotation angle; using the rotation quaternion between the first image and the second image as a parameter, the rotation matrix corresponding to the rotation quaternion is determined; based on the camera intrinsic parameter matrix and the rotation matrix, the homography matrix between the first image and the second image is determined; based on the quadratic form matrix and the homography matrix of each matched quadratic curve pair, a quadratic curve projection error function is constructed, and the direction vector transformation error function and the quadratic curve projection error function are weighted and fused to obtain a joint objective function;

[0047] For example, the projection error function of a conic section can be determined by the following formula:

[0048] ;

[0049]

[0050] in, This represents the projection error function of the quadratic curve. Indicates the number of pairs of matching quadratic curves; Let represent the quadratic form matrix of the quadratic curve matching the first image A in m; This represents the quadratic form matrix of the quadratic curve matching the second image B in m; Represents a rotation quaternion; This represents the homography matrix corresponding to the rotation quaternion; This represents the rotation matrix corresponding to the quaternion F; The inverse matrix representing the camera intrinsic parameter matrix; This represents the Frobenius norm.

[0051] For example, the joint objective function can be determined by the following formula:

[0052] ;

[0053] in, Describe the joint objective function; The weights represent the direction vector transformation error function; The weights represent the projection error function of the quadratic curve; This represents the direction vector transformation error function.

[0054] Using the unit module as a constraint, a Lagrange optimization function is constructed based on the joint objective function and the Lagrange multipliers; for example, the Lagrange optimization function can be determined by the following formula:

[0055] ;

[0056] in, Represent the Lagrange optimization function; Represents the Lagrange multipliers; This represents the transpose of a rotation quaternion.

[0057] The first partial derivative of the Lagrange optimization function with respect to the rotation quaternion and the second partial derivative with respect to the Lagrange multiplier are calculated respectively. Setting the first partial derivative to zero yields the first equation, and setting the second partial derivative to zero yields the second equation. Using the intermediate rotation quaternion as the initial value, the system of equations including the first and second equations is solved to obtain the target rotation quaternion. It should be noted that the camera pose can be determined using at least one existing technique based on the target rotation quaternion, and this invention does not limit this.

[0058] Optionally, extracting at least one pair of matching parallel lines and at least one pair of matching quadratic curves between the first image and the second image includes: detecting at least one rectangular object region and at least one circular object region in the first image, and detecting at least one rectangular object region and at least one circular object region in the second image; for each rectangular object region, detecting a pair of parallel lines in the rectangular object region; the pair of parallel lines includes two mutually parallel line segments; for each circular object region, detecting a quadratic curve in the circular object region; performing matching processing on the pair of parallel lines in the first image and the pair of parallel lines in the second image to obtain at least one pair of matching parallel lines between the first image and the second image; performing matching processing on the quadratic curves in the first image and the quadratic curves in the second image to obtain at least one pair of matching quadratic curves between the first image and the second image.

[0059] Here, a rectangular object region can be the image region corresponding to the detection box of a rectangular object. A circular object region can be the image region corresponding to the detection box of a circular or elliptical object.

[0060] Specifically, an object detection algorithm is used to detect objects in the first image, obtaining at least one rectangular object region and one circular object region in the first image; object detection is also performed on the second image, obtaining at least one rectangular object region and one circular object region in the second image; for each rectangular object region, a line segment detection algorithm is used to detect line segments within the rectangular object region, and line segments with a confidence level less than a set line segment confidence threshold are filtered out; for each pair of line segments, the angle between the direction vectors of the two line segments is calculated; if the angle is less than a preset angle threshold, the two line segments are determined as a pair of parallel lines; for each circular object region, a quadratic curve detection algorithm is used to detect quadratic curves within the circular object region, and quadratic curves with a confidence level less than a set quadratic curve confidence threshold are filtered out; candidate line pairs are constructed based on each pair of parallel line pairs in the first image and each pair of parallel line pairs in the second image; each candidate line pair includes one pair of parallel line pairs in the first image and one pair of parallel line pairs in the second image; for each candidate line pair, the LBD (Line Segment Difference) of the two line segments of the parallel line pairs in the first image within the candidate line pair is calculated. Using line segment geometric parameters such as the binary descriptor (BBD) descriptor, midpoint coordinates, length, and angle of the two line segments of the parallel line pairs in the second image, the similarity between the parallel line pairs in the first image and the parallel line pairs in the second image is calculated. Candidate line pairs with similarity higher than a first similarity threshold are identified as matching parallel line pairs. Candidate quadratic curve pairs are constructed based on the quadratic curves in the first image and the second image. Each candidate quadratic curve pair includes a quadratic curve in the first image and a quadratic curve in the second image. For each candidate quadratic curve pair, the similarity between the quadratic curves in the first image and the second image is calculated based on the center point coordinates, semi-major axis, semi-minor axis, and rotation angle of the quadratic curve in the first image and the same parameters as the quadratic curve in the second image. Candidate quadratic curve pairs with similarity higher than a second similarity threshold are identified as matching quadratic curve pairs.

[0061] Understandably, by adopting the above technical solution, the rectangular and circular object regions in the image are first detected, and then parallel line pairs and quadratic curves are extracted from the corresponding regions respectively. By identifying rectangular and circular regions, the detection range of lines and quadratic curves can be narrowed, which can reduce the amount of computation and reduce background noise interference. According to the different line categories, the similarity is determined by targeted parameters, which improves the accuracy of matching parallel line pairs and matching quadratic curve pairs.

[0062] Optionally, if no matching quadratic curve pair exists, the intermediate rotation quaternion is determined as the target rotation quaternion to determine the camera pose.

[0063] It is understandable that by adopting the above technical solution, when there is no matching quadratic curve pair, the intermediate rotation quaternion obtained by solving the parallel straight line pair is directly used as the final result to determine the output pose. This can ensure that the camera pose estimation method provided in this embodiment is normally usable in scenarios without quadratic curves, and can also avoid invalid calculations, thus taking into account both scene adaptability and running efficiency.

[0064] This invention constructs a direction vector transformation error function by matching parallel line pairs and solving the intermediate rotation quaternion to obtain a reliable initial solution for the rotation quaternion. In the case of a quadratic curve, the quadratic curve projection error is fused to form a joint objective function, and then the Lagrange multiplier is used to incorporate the unit modulus constraint for extreme value solving, thereby achieving complementarity between the two types of geometric constraints and improving the accuracy and robustness of camera pose estimation, especially in special scenarios such as sparse texture and varying illumination.

[0065] Example 2

[0066] Figure 2 This is a flowchart of a camera pose estimation method provided in Embodiment 2 of the present invention. The present invention is an optimization and improvement based on the technical solution of the above embodiments.

[0067] Furthermore, the phrase "using the rotation quaternion between the first image and the second image as a parameter, and constructing a direction vector conversion error function based on each of the matched parallel line pairs and the camera intrinsic matrix" is refined to: "For each matched parallel line pair, based on the homogeneous line vectors of the two lines in the parallel line pair of the first image in the matched parallel line pair, and the camera intrinsic matrix, determine the first unit direction vector corresponding to the matched parallel line pair; based on the homogeneous line vectors of the two lines in the parallel line pair of the second image in the matched parallel line pair, and the camera intrinsic matrix, determine the second unit direction vector corresponding to the matched parallel line pair; determine the quaternion of each first unit direction vector and the quaternion of each second unit direction vector; using the rotation quaternion between the first image and the second image as a parameter, and constructing a direction vector conversion error function based on the first quaternion and the second quaternion of each of the matched parallel line pairs."

[0068] It should be noted that for any parts not described in detail in the embodiments of the present invention, please refer to the description in the foregoing embodiments.

[0069] See Figure 2 The camera pose estimation method shown includes:

[0070] S201. Obtain the first and second images captured by the camera, as well as the camera intrinsic parameter matrix.

[0071] S202. Extract the matching pairs of parallel lines and matching pairs of quadratic curves between the first image and the second image.

[0072] S203. For each pair of matching parallel lines, determine the first unit direction vector corresponding to the pair of matching parallel lines based on the homogeneous line vectors of the two lines in the first image of the pair of matching parallel lines and the camera intrinsic parameter matrix.

[0073] S204. Based on the homogeneous line vectors of the two lines in the parallel line pair of the second image in the matched parallel line pair, and the camera intrinsic parameter matrix, determine the second unit direction vector corresponding to the matched parallel line pair.

[0074] S205. Determine the quaternion of each of the first unit direction vectors and the quaternion of each of the second unit direction vectors.

[0075] S206. Using the rotation quaternion between the first image and the second image as a parameter, construct a direction vector transformation error function based on the first and second quaternions of each of the matching parallel line pairs, and solve the problem with the goal of minimizing the function value of the direction vector transformation error function to obtain the intermediate rotation quaternion.

[0076] S207. If a pair of matching quadratic curves exists, then using the rotation quaternion between the first image and the second image as a parameter, a quadratic curve projection error function is constructed based on each of the matching quadratic curve pairs and the camera intrinsic parameter matrix. The direction vector transformation error function and the quadratic curve projection error function are then fused to obtain a joint objective function.

[0077] S208. Using the unit module as a constraint condition, construct a Lagrange optimization function based on the joint objective function and the Lagrange multipliers, and use the intermediate rotated quaternion as the initial value for iteration to solve for the extreme value of the Lagrange optimization function to obtain the target rotated quaternion.

[0078] S209. Determine the pose of the camera based on the target rotation quaternion.

[0079] In this embodiment, the first unit direction vector can be the unit direction vector in the first image; the second unit direction vector can be the unit direction vector in the second image.

[0080] Specifically, using a certain algorithm, the first unit direction vector corresponding to the matching parallel line pair is determined based on the homogeneous line vectors of the two lines in the first image of the matching parallel line pair and the camera intrinsic parameter matrix; the process for determining the second unit direction vector is similar to that for the first unit direction vector, and will not be repeated here; each first unit direction vector is converted into a corresponding quaternion to obtain the quaternion of each first unit direction vector, and each second unit direction vector is converted into a corresponding quaternion to obtain the quaternion of each second unit direction vector; using a certain algorithm, with the rotation quaternion between the first image and the second image as a parameter, a direction vector conversion error function is constructed based on the first quaternion and the second quaternion of each matching parallel line pair.

[0081] Optionally, determining the first unit direction vector corresponding to the matching parallel line pair based on the homogeneous line vectors of the two lines in the first image's parallel line pair and the camera intrinsic parameter matrix includes: determining the coordinates of the first disappearance point corresponding to the first image's parallel line pair based on the homogeneous line vectors of the two lines in the first image's parallel line pair; and determining the first unit direction vector corresponding to the matching parallel line pair based on the coordinates of the first disappearance point and the camera intrinsic parameter matrix.

[0082] Wherein, the coordinates of the first disappearance point can be the coordinates of the disappearance points of the parallel line pairs in the first image; for example, the coordinates of the first disappearance point can be determined by the following formula:

[0083] ;

[0084] in, This represents the coordinates of the first disappearing point of the first image A in the matching parallel line pair K; This represents a line segment that matches the first image A of the parallel line pair in the matching parallel line pair K; This represents the other line segment of the parallel line pair that matches the first image A in the parallel line pair K.

[0085] Based on the coordinates of the first disappearance point and the camera intrinsic parameter matrix, the first direction vector corresponding to the matching parallel line pair is determined; for example, the first direction vector can be determined by the following formula:

[0086] ;

[0087] in, This represents the first direction vector of the first image A in the pair of parallel lines K;

[0088] Normalizing the first direction vector yields a first unit direction vector; for example, the first unit direction vector can be determined by the following formula:

[0089] ;

[0090] in, This represents the first unit direction vector of the first image A in the pair of matching parallel lines K.

[0091] Optionally, the step of constructing a direction vector conversion error function using the rotation quaternion between the first image and the second image as a parameter, based on the first and second quaternions of each of the matching parallel line pairs, includes: for each matching parallel line pair, using the rotation quaternion between the first image and the second image as a parameter, constructing a direction vector conversion error sub-function corresponding to the matching parallel line pair based on the first and second quaternions of the matching parallel line pair; and fusing the direction vector conversion error sub-functions corresponding to each of the matching parallel line pairs to obtain the direction vector conversion error function.

[0092] Wherein, the first quaternion can be the transformation result of the first unit direction vector; the second quaternion can be the transformation result of the second unit direction vector. For example, the direction vector transformation error sub-function can be determined by the following formula:

[0093] ;

[0094] in, This represents the direction vector transformation error subfunction corresponding to the matching parallel line pair K; This represents the first quaternion corresponding to the pair of parallel lines K; This represents the second quaternion corresponding to the pair of parallel lines K; Represents the conjugate quaternion of a rotation quaternion.

[0095] For example, the direction vector transformation error function can be determined by the following formula:

[0096] ;

[0097] in, This indicates the number of pairs of matching parallel lines.

[0098] It is understandable that by adopting the above technical solution, the rotational consistency of the corresponding feature can be accurately measured by each sub-function, multiple directional constraints can be aggregated by fusing multiple sub-functions, and the global optimal solution of the rotation quaternion can be achieved by minimizing the overall error, thereby improving the stability of the intermediate rotation quaternion.

[0099] This invention, in its embodiment, solves for the unit direction vectors of the first and second images respectively by combining the homogeneous line vectors and camera intrinsic parameters for each pair of matched parallel lines, and then converts them into quaternion form to construct a function. This achieves the transformation of two-dimensional line features into three-dimensional direction constraints, and the error metric in quaternion form is consistent with the rotation parameters, ensuring the geometric consistency of the error definition and providing a rigorous target for the initial solution.

[0100] Example 3

[0101] Figure 3 This is a schematic diagram of a camera pose estimation device according to Embodiment 3 of the present invention. This embodiment of the present invention is applicable to situations involving camera pose estimation. The device can execute a camera pose estimation method and can be implemented in hardware and / or software. The device can be configured in an electronic device.

[0102] See Figure 3 The camera pose estimation device shown includes an acquisition module 301, an extraction module 302, a first solution module 303, a fusion module 304, a second solution module 305, and a pose determination module 306, wherein...

[0103] The acquisition module 301 is used to acquire a first image and a second image captured by the camera, as well as a camera intrinsic parameter matrix; the acquisition time of the second image is before that of the first image.

[0104] Extraction module 302 is used to extract at least one pair of matching parallel lines and at least one pair of matching quadratic curves between the first image and the second image; the pair of matching parallel lines includes a pair of parallel lines in the first image and a pair of parallel lines in the second image; the pair of matching quadratic curves includes a quadratic curve in the first image and a quadratic curve in the second image.

[0105] The first solving module 303 is used to construct a direction vector transformation error function based on the rotation quaternion between the first image and the second image, according to each of the matching parallel line pairs and the camera intrinsic parameter matrix, and to solve the problem with the goal of minimizing the function value of the direction vector transformation error function to obtain the intermediate rotation quaternion.

[0106] The fusion module 304 is used to construct a quadratic curve projection error function based on the rotation quaternion between the first image and the second image, using the rotation quaternion as a parameter, according to each of the matching quadratic curve pairs and the camera intrinsic parameter matrix, and to fuse the direction vector transformation error function and the quadratic curve projection error function to obtain a joint objective function if a matching quadratic curve pair exists.

[0107] The second solution module 305 is used to take the unit module as the constraint condition of the rotation quaternion, construct the Lagrange optimization function according to the joint objective function and the Lagrange multiplier, and use the intermediate rotation quaternion as the initial value of the iteration to solve the extreme value of the Lagrange optimization function to obtain the target rotation quaternion.

[0108] The pose determination module 306 is used to determine the pose of the camera based on the target rotation quaternion.

[0109] This invention constructs a direction vector transformation error function by matching parallel line pairs and solving the intermediate rotation quaternion to obtain a reliable initial solution for the rotation quaternion. In the case of a quadratic curve, the quadratic curve projection error is fused to form a joint objective function, and then the Lagrange multiplier is used to incorporate the unit modulus constraint for extreme value solving, thereby achieving complementarity between the two types of geometric constraints and improving the accuracy and robustness of camera pose estimation, especially in special scenarios such as sparse texture and varying illumination.

[0110] Optionally, the extraction module 302 is specifically used for:

[0111] Detect at least one rectangular object region and at least one circular object region in the first image, and detect at least one rectangular object region and at least one circular object region in the second image;

[0112] For each rectangular object region, detect the pairs of parallel lines in the rectangular object region; each pair of parallel lines includes two mutually parallel line segments.

[0113] For each circular object region, detect the quadratic curve within that region.

[0114] The parallel line pairs in the first image and the parallel line pairs in the second image are matched to obtain at least one matching parallel line pair between the first image and the second image.

[0115] The quadratic curves in the first image and the second image are matched to obtain at least one pair of matching quadratic curves between the first image and the second image.

[0116] Optionally, the first solver module 303 includes:

[0117] The first vector determination unit is used to determine the first unit direction vector corresponding to each matching parallel line pair based on the homogeneous line vectors of the two lines in the parallel line pair of the first image in the matching parallel line pair and the camera intrinsic parameter matrix.

[0118] The second vector determination unit is used to determine the second unit direction vector corresponding to the matching parallel line pair based on the homogeneous line vectors of the two lines in the parallel line pair of the second image in the matching parallel line pair, and the camera intrinsic parameter matrix.

[0119] The quaternion determination unit is used to determine the quaternion of each of the first unit direction vectors and the quaternion of each of the second unit direction vectors;

[0120] The construction unit is used to construct a direction vector transformation error function based on the first quaternion and the second quaternion of each of the matching parallel line pairs, using the rotation quaternion between the first image and the second image as a parameter.

[0121] Optionally, the first vector determining unit is specifically used for:

[0122] Based on the homogeneous line vectors of the two lines in the first image of the first image of the matching parallel line pair, determine the coordinates of the first disappearance point corresponding to the parallel line pair of the first image.

[0123] Based on the coordinates of the first disappearance point and the camera intrinsic parameter matrix, determine the first unit direction vector corresponding to the matching parallel line pair.

[0124] Optional, building blocks, specifically used for:

[0125] For each pair of matching parallel lines, using the rotation quaternion between the first image and the second image as a parameter, a direction vector transformation error subfunction corresponding to the pair of matching parallel lines is constructed based on the first quaternion and the second quaternion of the pair of matching parallel lines.

[0126] The direction vector conversion error function is obtained by fusing the direction vector conversion error sub-functions corresponding to each of the matching parallel line pairs.

[0127] Optionally, the device may also include:

[0128] The auxiliary pose determination module is used to determine the intermediate rotation quaternion as the target rotation quaternion if no matching quadratic curve pair exists, thereby determining the pose of the camera.

[0129] The camera pose estimation device provided in the embodiments of the present invention can execute the camera pose estimation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the camera pose estimation method.

[0130] Example 4

[0131] Figure 4A schematic diagram of a camera pose estimation device 410 that can be used to implement embodiments of the present invention is shown. The camera pose estimation device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The camera pose estimation device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0132] like Figure 4 As shown, the camera pose estimation device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 or a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 can also store various programs and data required for the operation of the camera pose estimation device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0133] Multiple components in the camera pose estimation device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, optical disk, etc.; and a communication unit 419, such as a network card, modem, wireless transceiver, etc. The communication unit 419 allows the camera pose estimation device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0134] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as camera pose estimation methods.

[0135] In some embodiments, the camera pose estimation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program may be loaded into and / or installed onto the camera pose estimation device 410 via ROM 412 and / or communication unit 419. When the computer program is loaded into RAM 413 and executed by processor 411, one or more steps of the camera pose estimation method described above may be performed. Alternatively, in other embodiments, processor 411 may be configured to perform the camera pose estimation method by any other suitable means (e.g., by means of firmware).

[0136] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0137] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable camera pose estimation device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0138] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0139] To provide user interaction, the systems and techniques described herein can be implemented on a camera pose estimation device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the camera pose estimation device. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0140] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0141] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability.

[0142] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0143] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for estimating the pose of a camera, characterized in that, The method includes: Acquire a first image and a second image captured by the camera, as well as a camera intrinsic parameter matrix; the second image was acquired before the first image. Extract matching pairs of parallel lines and matching pairs of quadratic curves between the first image and the second image; the matching pairs of parallel lines include a pair of parallel lines in the first image and a pair of parallel lines in the second image; the matching pairs of quadratic curves include a quadratic curve in the first image and a quadratic curve in the second image. Using the rotation quaternion between the first image and the second image as a parameter, a direction vector transformation error function is constructed based on each of the matching parallel line pairs and the camera intrinsic parameter matrix. The intermediate rotation quaternion is obtained by solving the problem with the goal of minimizing the function value of the direction vector transformation error function. If a pair of matching quadratic curves exists, then using the rotation quaternion between the first image and the second image as a parameter, a quadratic curve projection error function is constructed based on each pair of matching quadratic curves and the camera intrinsic parameter matrix. The direction vector transformation error function and the quadratic curve projection error function are then fused to obtain a joint objective function. Using the unit module as a constraint, a Lagrange optimization function is constructed based on the joint objective function and the Lagrange multipliers. The intermediate rotated quaternion is used as the initial value for iteration. The extreme value of the Lagrange optimization function is then solved to obtain the target rotated quaternion. The camera pose is determined based on the target rotation quaternion.

2. The method according to claim 1, characterized in that, The step of extracting at least one pair of matching parallel lines and at least one pair of matching quadratic curves between the first image and the second image includes: Detect at least one rectangular object region and at least one circular object region in the first image, and detect at least one rectangular object region and at least one circular object region in the second image; For each rectangular object region, detect the pairs of parallel lines in the rectangular object region; each pair of parallel lines includes two mutually parallel line segments. For each circular object region, detect the quadratic curve within that region. The parallel line pairs in the first image and the parallel line pairs in the second image are matched to obtain at least one matching parallel line pair between the first image and the second image. The quadratic curves in the first image and the second image are matched to obtain at least one pair of matching quadratic curves between the first image and the second image.

3. The method according to claim 1, characterized in that, The step of constructing a direction vector transformation error function using the rotation quaternion between the first image and the second image as a parameter, based on each of the matched parallel line pairs and the camera intrinsic parameter matrix, includes: For each pair of matching parallel lines, the first unit direction vector corresponding to the pair of matching parallel lines is determined based on the homogeneous line vectors of the two lines in the pair of parallel lines in the first image of the pair of matching parallel lines, and the camera intrinsic parameter matrix. Based on the homogeneous line vectors of the two lines in the parallel line pair of the second image in the matching parallel line pair, and the camera intrinsic parameter matrix, determine the second unit direction vector corresponding to the matching parallel line pair. Determine the quaternion of each first unit direction vector and the quaternion of each second unit direction vector; Using the rotation quaternion between the first image and the second image as a parameter, a direction vector transformation error function is constructed based on the first and second quaternions of each of the matching parallel line pairs.

4. The method according to claim 3, characterized in that, The step of determining the first unit direction vector corresponding to the matching parallel line pair based on the homogeneous line vectors of the two lines in the first image of the matching parallel line pair and the camera intrinsic parameter matrix includes: Based on the homogeneous line vectors of the two lines in the first image of the first image of the matching parallel line pair, determine the coordinates of the first disappearance point corresponding to the parallel line pair of the first image. Based on the coordinates of the first disappearance point and the camera intrinsic parameter matrix, determine the first unit direction vector corresponding to the matching parallel line pair.

5. The method according to claim 3, characterized in that, The step of constructing a direction vector transformation error function based on the rotation quaternion between the first image and the second image, using the first quaternion and the second quaternion of each of the matched parallel line pairs as parameters, includes: For each pair of matching parallel lines, using the rotation quaternion between the first image and the second image as a parameter, a direction vector transformation error subfunction corresponding to the pair of matching parallel lines is constructed based on the first quaternion and the second quaternion of the pair of matching parallel lines. The direction vector conversion error function is obtained by fusing the direction vector conversion error sub-functions corresponding to each of the matching parallel line pairs.

6. The method according to claim 1, characterized in that, The method further includes: If no matching quadratic curve pair exists, the intermediate rotation quaternion is determined as the target rotation quaternion to determine the camera pose.

7. A camera pose estimation device, characterized in that, The device includes: The acquisition module is used to acquire a first image and a second image captured by the camera, as well as a camera intrinsic parameter matrix; the acquisition time of the second image is before that of the first image. An extraction module is used to extract at least one pair of matching parallel lines and at least one pair of matching quadratic curves between the first image and the second image; the pair of matching parallel lines includes a pair of parallel lines in the first image and a pair of parallel lines in the second image; the pair of matching quadratic curves includes a quadratic curve in the first image and a quadratic curve in the second image. The first solution module is used to construct a direction vector transformation error function based on the rotation quaternion between the first image and the second image, according to each of the matching parallel line pairs and the camera intrinsic parameter matrix, and to solve the problem with the goal of minimizing the function value of the direction vector transformation error function to obtain the intermediate rotation quaternion. The fusion module is used to construct a quadratic curve projection error function based on the rotation quaternion between the first image and the second image, using the rotation quaternion as a parameter, according to each of the matching quadratic curve pairs and the camera intrinsic parameter matrix, and to fuse the direction vector transformation error function and the quadratic curve projection error function to obtain a joint objective function if a matching quadratic curve pair exists. The second solution module is used to take the unit module as a constraint condition for the modulus of the rotated quaternion, construct a Lagrange optimization function based on the joint objective function and the Lagrange multipliers, and use the intermediate rotated quaternion as the initial value for iteration to solve the extreme value of the Lagrange optimization function to obtain the target rotated quaternion. The pose determination module is used to determine the pose of the camera based on the target rotation quaternion.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a camera pose estimation method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the pose estimation method of the camera according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the pose estimation method for the camera according to any one of claims 1-6.

Images