Global optimal rotation estimation method for wide-field camera

Abstract

The invention discloses a global optimal rotation estimation method for a wide-field camera. A unified wide-view-field spherical projection model is defined and is suitable for any fisheye and panoramic camera; based on the projection model, a mathematical relationship between 2D-3D data observed by a fisheye camera and a panoramic camera can be simplified into a linear model relationship consistent with that of a traditional pinhole camera, and a consistent set maximization formula is further introduced based on the linear relationship for visual localization; and finally, providing a global optimal rotation search algorithm based on region overlapping judgment and acceleration binary angle interval voting to solve optimal rotation. The method is suitable for a wide-view-field camera model, is not limited by the problem that the view field of a traditional camera is narrow, and improves the success rate of visual positioning in challenging scenes such as long-term shielding and low texture. According to the method, the optimal rotation is solved based on geometric knowledge, and high efficiency and robustness can be always kept in a complex scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual positioning, and in particular relates to a global optimal rotation estimation method for a wide-field-of-view camera. Background Art

[0002] Visual localization involves using a query image to recover the camera position from a pre-built map. It is a core technology in fields such as autonomous driving, robotic navigation, and augmented reality. This technology faces challenges such as dynamic occlusion, low texture, and environmental variations. This research aims to overcome these challenges by introducing a wider field of view (FOV) and developing a more robust pose solution algorithm.

[0003] Traditional pinhole cameras, due to their narrow field of view, are limited in applications requiring extensive spatial perception. In contrast, panoramic and fisheye cameras capture a wider scene, mitigate the effects of dynamic occlusion, and provide rich texture information, which facilitates robust pose estimation. However, existing pinhole camera pose estimation algorithms fail to fully exploit the advantages of these wide-field-of-view images. While some algorithms, such as MLPnP, support fisheye camera images, they lack the ability to handle 360° viewing angles. Therefore, designing a unified PnP solution for panoramic and fisheye cameras is a key research direction.

[0004] Outliers are inevitable during feature extraction and matching, making robust outlier removal schemes crucial. Random Sample Consensus Algorithm (RANSAC) is widely used due to its simplicity and efficiency, but it can lead to local convergence issues, affecting positioning accuracy. Global optimization methods such as branch-and-bound (BnB) and mixed integer programming can guarantee optimal solutions, but their high computational complexity makes them difficult to meet the requirements of real-time or large-scale applications.

[0005] In previous work, the 3D rotation problem was simplified by gravity alignment, but this method relies on accurate gravity estimation, which limits its versatility. Therefore, improving the efficiency of the global optimal 2D-3D registration in the six-degree-of-freedom (6DoF) setting remains a challenge. In addition, the special structure of the SO(3) group makes the rotation parameters in the 3D rigid body transformation more difficult to solve than the translation parameters. Once the rotation problem is solved, the translation parameters are relatively easy to determine. Therefore, developing robust 3D rotation estimation methods applicable to various scenarios remains an important task.

[0006] In order to solve the above problems, the present invention proposes a global optimal rotation estimation method for wide field of view cameras. Summary of the Invention

[0007] This invention aims to overcome the shortcomings of existing technologies by providing a globally optimal rotation estimation method for wide-field-of-view cameras. By unifying a universal wide-field-of-view spherical camera model, the mathematical relationship between 2D-3D data observed by fisheye and panoramic cameras is simplified to a linear relationship similar to that of traditional pinhole cameras. Based on this linear relationship, the problem is transformed into a visual localization method based on a maximum consistent set. The optimal rotation estimate is then solved using a proposed optimal algorithm for global optimal rotation search.

[0008] In a first aspect, the present invention provides a global optimal rotation estimation method for a wide field of view camera, comprising the following steps:

[0009] S1: Define a general spherical projection model to accommodate various wide-field-of-view cameras;

[0010] S2: Applying the general spherical projection model to the panoramic camera model;

[0011] S3: Applying the general spherical projection model to the fisheye camera model;

[0012] S4: Based on the model defined in step S1, the mathematical relationship between the 2D-3D data observed by the fisheye camera and the panoramic camera is converted into a linear relationship, and the consistent set maximization formula is introduced to perform visual positioning;

[0013] S5: A global optimal rotation search method based on region overlap judgment and accelerated binary angle interval voting is proposed to solve the maximum consistent set of rotation problem, and then the global optimal rotation is solved; according to the solution results, the optimal rotation posture parameters of the wide field of view camera are obtained.

[0014] Furthermore, step S1 specifically includes:

[0015] For the kth 2D-3D visual measurement, given a 3D point in the world and a 2D pixel In an absolutely ideal situation, there is a projection expression like this:

[0016] u k =π(Rp k +t,K,d), (1)

[0017] in, is the obtained 2D-3D visual observation set, R∈SO(3) and is the camera pose to be estimated, π represents the projection function, and d is the distortion coefficient;

[0018] The general spherical projection model is: 2D points on the imaging plane are uniformly projected onto a unit sphere emanating from the optical center of the camera, and then the general mathematical relationship between the projection vector on this unit sphere and the 3D point is used to perform subsequent pose derivation; that is, the spherical projection surface is represented as Θ, and the vector projecting the 2D point onto the spherical surface Θ is represented as The normalized projection vector is recorded as

[0019] Furthermore, step S2 specifically includes:

[0020] In a panoramic camera, the camera coordinates are expressed in a spherical coordinate system, where lon and lat represent the longitude and latitude of the camera, respectively, subject to the constraints lon∈(-π,π) and lat∈(-π / 2,π / 2). These coordinates are then projected onto the two-dimensional image plane using a cylindrical equidistant projection method. If H and W represent the height and width of the image, respectively, then ideally, the camera's intrinsic parameter K is determined by specifying the image size (H,W). The mathematical expression is as follows:

[0021]

[0022] Then, the pixel coordinates are lifted to the imaging plane, and we have:

[0023]

[0024] Finally, based on the back-projection process, we get:

[0025]

[0026] Furthermore, step S3 specifically includes:

[0027] The fisheye camera model introduces an offset parameter ∈ in the projection process to correct the nonlinear distortion of the wide-angle lens;

[0028] First, the pixel coordinates are converted to the imaging plane using the camera's intrinsic parameter K:

[0029]

[0030] Then, according to the following distortion equation:

[0031] x(1+k1(x 2 +y 2 )+k2(x 2 +y 2 ) 2 )=x0

[0032] y(1+k1(x 2 +y2 )+k2(x 2 +y 2 ) 2 )=y0(6)

[0033] Use the Gauss-Newton method to find the approximate solution for the x and y parameters; where the point [x,y,z] T is the undistorted point on the z-normalized imaging plane, T represents the transpose operation; at the same time, this point is consistent with the point obtained by normalizing the z-axis coordinate of the projection vector v after applying an offset ∈ along the z-axis, so:

[0034]

[0035] Among them, r represents the vector The module length, Applying an offset to the z-axis of v and normalizing the z-axis yields:

[0036]

[0037] Then, the normalized spherical coordinates are:

[0038]

[0039] in, It is ordered by Substitution Got it.

[0040] Furthermore, step S4 specifically includes:

[0041] Based on the derivation in Equation (4) and Equation (9), the 2D-3D correspondence constraint in Equation (1) is reformulated as the following collinearity constraint, which is mathematically expressed as:

[0042] s k v k =Rp k +t (10)

[0043] where s k represents an unknown scaling factor related to depth; this linear relationship is consistent with the 2D-3D visual observation set observed in a traditional pinhole camera; therefore, the following subscript set of corresponding relations is defined:

[0044]

[0045] According to the derivation in step S1, v k Represents u k The corresponding projection vector, Represents the total number of visual observation set correspondences; thus far, the visual localization problem of wide - field cameras such as fisheye and panoramic cameras has been transformed into a 2D - 3D visual localization problem similar to that of traditional pinhole cameras; next, based on this visual observation set Pose estimation is performed. Since the occurrence of outliers in the visual observation set is inevitable, the constraint equation (10) is further expressed as the following mathematical model:

[0046] s k v k = Rp k + t + o<00了0023>+ e k (12) <00002几1>[[ID=侣]]Where |e k | < n represents bounded random measurement noise, o k = 0 if and only if k is an inlier, and outliers have arbitrary unbounded values; also, due to the presence of outliers, directly reducing the reprojection error may lead to serious errors; therefore, a consensus set maximization formula is introduced, which finds the set of inliers with the largest number by searching for the optimal rotation matrix and translation; the formula is as follows:

[0048]

[0049] subject to |s k v k -(Rp k + t)| ≤ n, (13)

[0050] Where, is a subset of, representing the set of all elements that satisfy the reprojection error less than n, that is, inliers; when only considering the pure rotation problem, equation (12) can be simplified to:<000028乙>

[0051] s k v k = Rp k + o k + e k (14)

[0052] Therefore, for inliers, it should satisfy s k v k - Rp [[ID=6止]] k ≤ n; then, the error metric is converted from Euclidean distance to angular distance:

[0053] ∠(q<创000038> ,Rp k ) ≤ γ k , (15)

[0054] where q k =s k v k ; In addition, the inlier threshold γ k can be Then, as a specialization of formula (13), the maximum consistent set expression related only to rotation is as follows:

[0055]

[0056] subject to|q k -Rp k |≤γ k , (16).

[0057] Furthermore, step S5 specifically includes:

[0058] A global optimal rotation search method based on region overlap judgment and accelerated binary angle interval voting is introduced to solve Equation (16). The method includes: first, converting the problem of finding the maximum consistent set into the judgment of pairwise consensus; second, simplifying the judgment process of pairwise consensus by analyzing the angle between vectors instead of directly aligning the vectors; finally, using the rotation dimensionality reduction method, further simplifying this judgment from three-dimensional complexity to one-dimensional angle evaluation;

[0059] The rotation of the measured value k that satisfies formula (15) is denoted as R k , and decompose it into R k =AB; where B is determined by formula (15), A is around the axis Ba k A rotation of θ∈[-π,π] angle; since A maintains Ba k unchanged, so the constraint (15) of the measurement value k is always satisfied; then the existence of rotation A is to make R k Can also be a i Go to b i ; will make a k Perfectly aligned b k The rotation is defined as Then there is In addition, define a region It represents the measured value b k At the angular distance γ k A spherical area formed by rotating within the range; definition A θ,r , which represents a rotation about the axis r by an angle θ; the possible region obtained by applying this rotation A to any unit normal point p is denoted as circ(p,r):

[0060] circ(p,r):={Aθ,r p|θ∈[-π,π]} (16)

[0061] Then based on the above R k From the decomposition of k is the axis of rotation of rotation A, then The interior is the set of possible rotation axes of the rotation transformation A, so for any i≠k, R k a i The set of regions can be expressed as follows:

[0062]

[0063] Therefore, if and only if When k and i are in pairwise consensus; based on this intersection condition, all measurements i that cannot be aligned with k can be filtered out; then, under the condition that k and i are in pairwise consensus, it is further found that the interval θ∈[-π,π] that guarantees the intersection can be further narrowed; for a fixed θ, for all and R k Put a i The possible position to which it can be rotated is R k a i , these locations are in the area:

[0064]

[0065] Where δ(θ)∶=2|θ|sin(γ k / 2)+γ k ; For any i≠k, if With L k (a i ) intersect, then ∠(A θ,r Ba i , b i )≤γ i The boundary of θ is According to the geometric relationship, the corresponding θ interval is solved;

[0066] For all bounded angle intervals corresponding to the obtained pairwise consensus, an accelerated voting algorithm based on a large binary table is used to solve the optimal rotation.

[0067] Furthermore, the accelerated voting algorithm is specifically as follows: first, the angle axis is discretized to speed up the search speed, then a large binary table is used to record the angle intervals that meet the conditions, and finally the votes of each interval are counted to obtain the highest voting interval, that is, the global optimal rotation is solved.

[0068] In a second aspect, the present invention provides a global optimal rotation estimation device for a wide field of view camera, comprising:

[0069] Power module, used to support hardware power supply;

[0070] Motion module, used to support the movement of the robot;

[0071] The acquisition module is used to obtain the query image of the robot's current posture and obtain the first K reference images from the 2D image database;

[0072] The matching module is used to calculate the feature points of the query image and the reference image, and match the feature points calculated in the query image with the feature points of the reference image to obtain m pairs of 3D-2D feature matching points;

[0073] The general spherical model module is used to define a spherical projection model that is common to wide-field-of-view cameras such as fisheye and panoramic cameras.

[0074] The global optimal rotation search module is used to obtain the globally optimal rotation estimation based on pairwise consensus judgment and accelerated binary voting.

[0075] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the global optimal rotation estimation method for a wide-field-of-view camera.

[0076] In a fourth aspect, the present invention provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the global optimal rotation estimation method for a wide field of view camera is implemented.

[0077] Compared with the prior art, the present invention has the following beneficial effects:

[0078] (1) This paper derives a universal spherical model with wide viewing angles that is applicable to any fisheye and panoramic cameras, thereby improving the probability of successfully solving poses in challenging scenes caused by long-term occlusion and low texture in visual localization tasks.

[0079] (2) The present invention proposes a global optimal rotation search module, which converts vector alignment into vector angle and uses an accelerated binary voting algorithm to vote on the diagonal interval to efficiently solve the deterministic optimal rotation estimate, thereby significantly improving the robustness of the visual positioning task in long-term changing scenes.

[0080] (3) The method proposed in the present invention can always maintain robust and efficient optimal rotation estimation in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1Schematic diagram of a global optimal rotation estimation method for a wide field of view camera according to the present invention;

[0082] Figure 2 This is a flow chart of the module relationships in the present invention. DETAILED DESCRIPTION

[0083] In response to the problems of limited field of view, frequent outliers and insufficient algorithm robustness of traditional pinhole cameras in the process of visual positioning, the present invention proposes a global optimal rotation estimation method suitable for wide-field-of-view cameras for practical application scenarios such as autonomous driving and robot perception. This method realizes unified modeling of fisheye and panoramic cameras by constructing a unified wide-field-of-view spherical projection model. To achieve highly robust rotation estimation, the present invention further introduces a geometric optimization strategy based on regional overlap judgment, and combines it with an accelerated binary angle interval voting algorithm to effectively improve the efficiency of global rotation search. This method not only breaks through the problem of local optimality of traditional RANSAC, but also overcomes the limitation of poor real-time performance of global optimization methods. It can maintain efficient and accurate pose estimation performance in complex scenarios such as dynamic occlusion, low texture and long-term environmental changes, and has good engineering application potential and promotion value.

[0084] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific implementation methods.

[0085] Figure 1 The present invention discloses a global optimal rotation estimation method for a wide field of view camera, comprising the following steps:

[0086] S1: Define a general spherical projection model to accommodate various wide-field-of-view cameras.

[0087] For the traditional projection model, the present invention first processes the retrieved map image and query image using SuperPoint feature extraction and KNN matching to obtain the 2D-2D data association between the two. In terms of map construction, the scene points are triangulated using multi-frame observations and the reconstruction is refined through bundle adjustment. Finally, a set of 2D-3D visual measurements is obtained. For the kth 2D-3D visual measurement, given a 3D point in the world and a 2D pixel In an absolutely ideal situation (i.e., without considering factors such as noise), there is such a projection expression:

[0088] u k =π(Rp k +t,K,d), (1)

[0089] in, is the obtained 2D-3D visual observation set, R∈SO(3) and is the camera pose to be estimated. π represents the projection function and d is the distortion coefficient, which varies depending on the camera type.

[0090] Therefore, in order to unify the projection models of different wide-field cameras such as fisheye cameras and panoramic cameras, this paper proposes a universal spherical projection model. Specifically, the 2D points on the imaging plane are uniformly projected onto a unit sphere emanating from the optical center of the camera, and then the general mathematical relationship between the projection vector on this unit sphere and the 3D point is used to perform subsequent pose derivation. Here, for the convenience of subsequent description, the spherical projection surface is represented as Θ, and the vector projecting the 2D point onto the spherical surface Θ is represented as The normalized projection vector is recorded as

[0091] S2: Apply the general spherical projection model to the panoramic camera model.

[0092] In panoramic cameras, camera coordinates are typically expressed using a spherical coordinate system. lon and lat represent the camera's longitude and latitude, respectively, subject to the constraints lon∈(-π,π) and lat∈(-π / 2,π / 2). These coordinates are then projected onto the two-dimensional image plane, typically using a cylindrical equidistant projection method. If H and W represent the image's height and width, respectively, then ideally, the camera's intrinsic parameter K can be determined by specifying the image's dimensions (H,W). The mathematical expression is as follows:

[0093]

[0094] Then, the pixel coordinates are lifted to the imaging plane, and we have:

[0095]

[0096] Finally, based on the back-projection process, we can get:

[0097]

[0098] S3: Apply the general spherical projection model to the fisheye camera model.

[0099] The common fisheye camera projection process involves mapping 3D point coordinates from Ω to normalized spherical coordinates in Θ, then projecting them onto the imaging plane and applying a lens distortion model, ultimately converting them to pixel coordinates in Ψ. This paper uses the popular MEI fisheye camera model as an example to explain the projection process. The core of the MEI fisheye camera model is the introduction of an offset parameter ∈ (typically corresponding to the distance from the lens focal point to the projection sphere) during the projection process to correct for the nonlinear distortion of wide-angle lenses.

[0100] The specific process is as follows: First, the pixel coordinates are converted into the imaging plane using the camera's intrinsic parameter K:

[0101]

[0102] Then, according to the following distortion equation:

[0103] x(1+k1(x 2 +y 2 )+k2(x 2 +y 2 ) 2 )=x0

[0104] y(1+k1(x 2 +y 2 )+k2(x 2 +y 2 ) 2 )=y0(6)

[0105] The Gauss-Newton method can be used to find an approximate solution for the x and y parameters. T is the undistorted point on the z-normalized imaging plane, and T represents the transpose operation. At the same time, this point is consistent with the point obtained by normalizing the z-axis coordinate of the projection vector v after applying an offset ∈ along the z-axis. Specifically, we have:

[0106]

[0107] Among them, r represents the vector The module length, Then, applying an offset to the z-axis of v and normalizing the z-axis yields:

[0108]

[0109] Then, the normalized spherical coordinates can be solved as:

[0110]

[0111] Among them, λ is a simplified expression for subsequent derivation and is recorded as an intermediate variable. It is ordered by Substitution obtained

[0112] S4: Based on the model defined in step S1, transform the mathematical relationship between the 2D-3D data observed by the fisheye and panoramic cameras into a linear relationship, and introduce a consensus set maximization formula for visual localization.

[0113] Therefore, based on the derivations in equations (4) and (9), the 2D-3D correspondence constraint in equation (1) can be reformulated as the following collinearity constraint, mathematically expressed as:

[0114] s k v k = Rp k + t (10)

[0115] where s k represents an unknown scale factor related to depth. This linear relationship is consistent with the 2D-3D visual observation set observed in a traditional pinhole camera. Therefore, the following index set of corresponding relationships can be defined:

[0116]

[0117] According to the derivation in S1, v k represents the projection vector corresponding to u k , denotes the total number of corresponding relationships in the visual observation set. So far, the present invention has transformed the visual localization problem of wide-field cameras such as fisheye and panoramic cameras into a 2D-3D visual localization problem similar to that of a traditional pinhole camera. Next, the present invention performs pose estimation based on this visual observation set . Since the appearance of outliers in the visual observation set is inevitable, the constraint equation (10) can be further expressed as the following mathematical model:

[0118] s k v k = Rp k + t + o k + e k (12)<000046..1>

[0119] where |e k |< n represents bounded random measurement noise, o k = 0 if and only if k is an inlier, while outliers have arbitrary unbounded values. Also, due to the presence of outliers, directly reducing the reprojection error may lead to serious errors. Therefore, the present invention introduces a consensus set maximization formula, which finds the largest set of inliers by searching for the optimal rotation matrix and translation. The formula is as follows:

[0120]

[0121] subject to|s k v k -(Rp k +t)|≤n, (13)

[0122] in, yes The subset of represents the set of all elements that satisfy the reprojection error less than n, that is, the interior point. When only considering the pure rotation problem, Equation (12) can be simplified to:

[0123] s k v k =Rp k +o k +e k (14)

[0124] Therefore, for the interior point, it should satisfy s k v k -Rp k ≤n. ​​The error metric can then be converted from Euclidean distance to angular distance:

[0125] ∠(q k , Rp k )≤γ k , (15)

[0126] where q k =s k v k In addition, the inlier threshold γ k (in radians) can be obtained by Then, as a specialization of formula (13), the maximum consistent set expression related only to rotation is as follows:

[0127]

[0128] subject to|q k -Rp k |≤γ k , (16)

[0129] S5: A global optimal rotation search method based on region overlap judgment and accelerated binary angle interval voting is proposed to solve the problem of maximum consistent set of rotations, and then the global optimal rotation is solved.

[0130] The present invention introduces a global optimal rotation search method to solve Equation (16). The core steps of this method include: first, converting the problem of finding the largest consistent set into the judgment of pairwise consensus; then, simplifying the judgment process of pairwise consensus by analyzing the angle between vectors instead of directly aligning the vectors; finally, using the rotation dimensionality reduction method, further simplifying this judgment from three-dimensional complexity to one-dimensional angle evaluation.

[0131] Specifically, first, the rotation of the measurement value k that satisfies formula (15) is recorded as R k , and decompose it into R k =AB. Where B is determined by formula (15), A is around the axis Ba k A rotation of θ∈[-π,π]. Since A keeps Ba k is unchanged, so the constraint (15) of the measurement value k is always satisfied. Then the existence of rotation A is to make R k Can also be a i Go to b i In addition, it will make a k Perfectly aligned b k The rotation is defined as Then there is In addition, define a region It represents the measured value b k At the angular distance γ k A spherical area formed by rotating within the range. Definition A θ,r , which represents a rotation around the axis r by an angle θ. The possible region obtained by applying this rotation A to any unit normal point p is denoted as circ(p,r):

[0132] circ(p,r):={A θ,r p|θ∈[-π,π]} (16)

[0133] Then based on the above R k From the decomposition of k is the axis of rotation of rotation A, then The interior is the set of possible rotation axes of the rotation transformation A, so for any i≠k, R k a i The set of regions can be expressed as follows:

[0134]

[0135] Therefore, if and only if When k and i are in pairwise consensus. Based on this intersection condition, all measurements i that cannot be aligned with k can be filtered. Then, under the condition that k and i are in pairwise consensus, it is further found that the interval θ∈[-π,π] that guarantees the intersection can be further narrowed. Specifically, for a fixed θ, for all and R k Put a i The possible position to which it can be rotated is R k a i , these locations are in the area:

[0136]

[0137] Where δ(θ)∶=2|θ|sin(γ k / 2)+γ k For any i≠k, if With L k (a i ) intersect, then ∠(A θ,r Ba i , b i )≤γ i The boundary of θ is According to the geometric relationship, the corresponding θ interval can be solved.

[0138] For all bounded angle intervals corresponding to the obtained pairwise consensus, the present invention uses an accelerated voting algorithm based on a large binary table to find the optimal rotation. The details of this accelerated algorithm are as follows: first, the angle axis is discretized to speed up the search. Then, a large binary table is used to record the angle intervals that meet the conditions. Finally, the votes for each interval are counted to obtain the interval with the highest vote. This means that the globally optimal rotation is found.

[0139] According to the solution of the global optimal rotation, the optimal rotation attitude parameters of the wide-field camera can be solved stably and efficiently under the condition of a large number of abnormal matching points. The solution can be used as a key module in the visual positioning system, and can be directly used in 6DoF pose estimation tasks in the fields of autonomous driving, robot navigation, augmented reality, etc., thereby realizing accurate recovery of the camera's spatial position and orientation. In addition, this method can still achieve stable solution when there are only a very small number of inliers (such as 95% of the matching points are outliers), and has strong adaptability to camera types, can be adapted to a variety of wide-angle lens configurations, and has good industrial versatility and promotion potential. Therefore, the present invention has significant practical value and engineering application prospects in actual visual positioning systems under complex environments.

[0140] The present invention also discloses a device for globally optimal rotation estimation for a wide-field-of-view camera. The device includes a power supply unit, a motion unit, a camera unit, and a processing unit. The power supply unit supplies power to the other units. The motion unit supports the movement of a robot. The camera unit is used to build a two-dimensional image database and obtain the current image of the robot. The processing unit also includes a processor, a memory, and a program stored in the memory. When executed by the processor, the computer program implements the steps of a method for globally optimal rotation estimation for a wide-field-of-view camera.

[0141] The steps of obtaining the current query image of the robot from the camera unit; matching the feature points of the current query image with the reference image in the database; calculating the robot pose based on the matched feature points; and checking and optimizing based on the calculated pose are all part of the algorithm and are completed on a computer or a processor carried by the robot.

[0142] The method in this embodiment and the device in the foregoing embodiment are two aspects based on the same inventive concept. Those skilled in the art can clearly understand the system structure and implementation process in this embodiment based on the foregoing description.

[0143] For the convenience of description, when describing the above device, the functions are divided into various modules for separate description. The module relationship flow chart is as follows: Figure 2 As shown. Of course, when implementing the present invention, the functions of each module can be implemented in the same or multiple software or hardware. The specific functions of each module are as follows:

[0144] Power module, used to support hardware power supply;

[0145] Motion module, used to support the movement of the robot;

[0146] The acquisition module is used to obtain the query image of the robot's current posture and obtain the first K reference images from the 2D image database;

[0147] The matching module is used to calculate the feature points of the query image and the reference image, and match the feature points calculated in the query image with the feature points of the reference image to obtain m pairs of 3D-2D feature matching points;

[0148] The general spherical model module is used to define a spherical projection model that is common to wide-field-of-view cameras such as fisheye and panoramic cameras.

[0149] The global optimal rotation search module is used to obtain the globally optimal rotation estimation based on pairwise consensus judgment and accelerated binary voting.

[0150] The software product can be stored in a variety of storage media, such as ROM / RAM, disk, etc. It contains programs that enable a computer device to execute the methods described in various embodiments or parts of the embodiments of the present invention. These computer devices can be personal computers, servers, or MCUs, etc.

[0151] The device embodiments described are for illustrative purposes only, and the modules or units described as separate components may be physically separate or integrated. The components described as modules or units may or may not be physical modules and may be distributed across multiple modules. Depending on actual needs, some or all of the units may be selected to achieve the purpose of the embodiment.

[0152] Those skilled in the art can understand and implement the embodiment without any creative work, which means that they can make appropriate selections and configurations based on the described content and actual needs to achieve the purpose of the embodiment.

[0153] The present invention is applicable to a wide range of general-purpose and special-purpose computing system environments or configurations. These environments include, but are not limited to, personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above. In short, the present invention is applicable to a wide range of computing system environments or configurations, from personal devices to mainframe computers and distributed computing environments.

[0154] The above embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and replacements made by technicians in this field on the basis of the present invention without departing from the spirit and scope of the technical solution of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. A global optimal rotation estimation method for wide field of view cameras, characterized in that The following steps are involved: S1: Define a general spherical projection model to accommodate various wide-field-of-view cameras; S2: Applying the general spherical projection model to the panoramic camera model; S3: Applying the general spherical projection model to the fisheye camera model; S4: Based on the model defined in step S1, the mathematical relationship between the 2D-3D data observed by the fisheye camera and the panoramic camera is converted into a linear relationship, and the consistent set maximization formula is introduced to perform visual positioning; S5: A global optimal rotation search method based on region overlap judgment and accelerated binary angle interval voting is proposed to solve the maximum consistent set of rotations problem, and then the global optimal rotation is solved; According to the solution results, the optimal rotation posture parameters of the wide field of view camera are obtained.

2. The global optimal rotation estimation method for a wide field of view camera according to claim 1, wherein: Step S1 specifically includes: For the kth 2D-3D visual measurement, given a 3D point in the world and a 2D pixel In an absolutely ideal situation, there is a projection expression like this: in, is the obtained 2D-3D visual observation set, R∈SO(3) and is the camera pose to be estimated, π represents the projection function, and d is the distortion coefficient; The general spherical projection model is: 2D points on the imaging plane are uniformly projected onto a unit sphere emanating from the optical center of the camera, and then the general mathematical relationship between the projection vector on this unit sphere and the 3D point is used to perform subsequent pose derivation; that is, the spherical projection surface is represented as Θ, and the vector projecting the 2D point onto the spherical surface Θ is represented as The normalized projection vector is recorded as 3. The global optimal rotation estimation method for a wide field of view camera according to claim 2, wherein: Step S2 specifically includes: In a panoramic camera, the camera coordinates are expressed in a spherical coordinate system, where lon and lat represent the longitude and latitude of the camera, respectively, subject to the constraints lon∈(-π,π) and lat∈(-π / 2,π / 2). These coordinates are then projected onto the two-dimensional image plane using a cylindrical equidistant projection method. If H and W represent the height and width of the image, respectively, then ideally, the camera's intrinsic parameter K is determined by specifying the image size (H,W). The mathematical expression is as follows: Then, the pixel coordinates are lifted to the imaging plane, and we have: Finally, based on the back-projection process, we get:

4. The global optimal rotation estimation method for a wide field of view camera according to claim 3, wherein: Step S3 specifically includes: The fisheye camera model introduces an offset parameter ∈ in the projection process to correct the nonlinear distortion of the wide-angle lens; First, the pixel coordinates are converted to the imaging plane using the camera's intrinsic parameter K: Then, according to the following distortion equation: x(1+k1(x 2 +y 2 )+k2(x 2 +y 2 ) 2 )=x0 y(1+k1(x 2 +y 2 )+k2(x 2 +y 2 ) 2 )=y0(6) Use the Gauss-Newton method to find the approximate solution for the x and y parameters; where the point [x,y,z] T is the undistorted point on the z-normalized imaging plane, T represents the transpose operation; at the same time, this point is consistent with the point obtained by normalizing the z-axis coordinate of the projection vector v after applying an offset ∈ along the z-axis, so: Among them, r represents the vector The module length, Applying an offset to the z-axis of v and normalizing the z-axis yields: Then, the normalized spherical coordinates are: in, It is ordered by Substitution Got it.

5. The global optimal rotation estimation method for a wide field of view camera according to claim 4, wherein: Step S4 specifically includes: Based on the derivation in Equation (4) and Equation (9), the 2D-3D correspondence constraint in Equation (1) is reformulated as the following collinearity constraint, which is mathematically expressed as: s k v k =Rp k +t (10) where s k represents an unknown scaling factor related to depth; this linear relationship is consistent with the 2D-3D visual observation set observed in a traditional pinhole camera; therefore, the following subscript set of corresponding relations is defined: According to the derivation in step S1, v k Represents u k The corresponding projection vector, Represents the total number of correspondences in the visual observation set; So far, the visual positioning problem of wide-field cameras such as fisheye and panoramic cameras has been transformed into a 2D-3D visual positioning problem similar to that of traditional pinhole cameras; Next, based on the visual observation set For pose estimation, since the appearance of outliers in the visual observation set is inevitable, the constraint formula (10) is further expressed as the following mathematical model: s k v k =Rp k +t+o k +e k (12) where |e k | <n represents the bounded random measurement noise, o k = 0 if and only if k is an inlier, while outliers have arbitrary unbounded values; also, due to the presence of outliers, directly reducing the reprojection error may lead to serious errors; therefore, a consensus set maximization formula is introduced, which finds the set of inliers with the largest number by searching for the optimal rotation matrix and translation; the formula is as follows: in, yes The subset of represents the set of all elements that satisfy the reprojection error less than n, that is, the interior point; when only considering the pure rotation problem, Equation (12) can be simplified to: s k v k =Rp k +o k +e k (14) Therefore, for the interior point, it should satisfy s k v k -Rp k ≤n; then, convert the error metric from Euclidean distance to angular distance: where q k =s k v k ; In addition, the inlier threshold γ k can be Then, as a specialization of formula (13), the maximum consistent set expression related only to rotation is as follows:

6. The global optimal rotation estimation method for a wide field of view camera according to claim 5, wherein: Step S5 specifically includes: A global optimal rotation search method based on region overlap judgment and accelerated binary angle interval voting is introduced to solve Equation (16). The method includes: first, converting the problem of finding the maximum consistent set into the judgment of pairwise consensus; second, simplifying the judgment process of pairwise consensus by analyzing the angle between vectors instead of directly aligning the vectors; finally, using the rotation dimensionality reduction method, further simplifying this judgment from three-dimensional complexity to one-dimensional angle evaluation; The rotation of the measured value k that satisfies formula (15) is denoted as R k , and decompose it into R k =AB; where B is determined by formula (15), A is around the axis Ba k A rotation of θ∈[-π,π] angle; since A maintains Ba k unchanged, so the constraint (15) of the measurement value k is always satisfied; then the existence of rotation A is to make R k Can also be a i Go to b i ; will make a k Perfectly aligned b k The rotation is defined as Then there is In addition, define a region It represents the measured value b k At the angular distance γ k A spherical area formed by rotating within the range; definition A θ,r , which represents a rotation about the axis r by an angle θ; the possible region obtained by applying this rotation A to any unit normal point p is denoted as circ(p,r): circ(p,r) ∶={A θ,r p | θ ∈ [−π,π]} (16) Then based on the above R k From the decomposition of k is the axis of rotation of rotation A, then The interior is the set of possible rotation axes of the rotation transformation A, so for any i≠k, R k a i The set of regions can be expressed as follows: Therefore, if and only if When k and i are in pairwise consensus; based on this intersection condition, all measurements i that cannot be aligned with k can be filtered out; then, under the condition that k and i are in pairwise consensus, it is further found that the interval θ∈[-π,π] that guarantees the intersection can be further narrowed; for a fixed θ, for all and R k Put a i The possible position to which it can be rotated is R k a i , these locations are in the area: Where δ(θ)∶=2|θ|sin(γ k / 2)+γ k ; For any i≠k, if With L k (a i ) intersect, then ∠(A θ,r Ba i , b i )≤γ i The boundary of θ is According to the geometric relationship, the corresponding θ interval is solved; For all bounded angle intervals corresponding to the obtained pairwise consensus, an accelerated voting algorithm based on a large binary table is used to solve the optimal rotation.

7. The global optimal rotation estimation method for a wide field of view camera according to claim 6, wherein: The accelerated voting algorithm is specifically as follows: first, the angle axis is discretized to speed up the search, then a large binary table is used to record the angle intervals that meet the conditions, and finally the votes of each interval are counted to obtain the highest voting interval, that is, the global optimal rotation is solved.

8. A global optimal rotation estimation device for a wide field of view camera implementing the method according to any one of claims 1 to 7, characterized in that: include: Power module, used to support hardware power supply; Motion module, used to support the movement of the robot; The acquisition module is used to obtain the query image of the robot's current posture and obtain the first K reference images from the 2D image database; The matching module is used to calculate the feature points of the query image and the reference image, and match the feature points calculated in the query image with the feature points of the reference image to obtain m pairs of 3D-2D feature matching points; The general spherical model module is used to define a spherical projection model that is common to wide-field-of-view cameras such as fisheye and panoramic cameras. The global optimal rotation search module is used to obtain the globally optimal rotation estimation based on pairwise consensus judgment and accelerated binary voting.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed in a computer, the computer is caused to execute the global optimal rotation estimation method for a wide-field-of-view camera according to any one of claims 1 to 7.

10. A computing device comprising a memory and a processor, characterized in that: The memory stores executable code, and when the processor executes the executable code, the global optimal rotation estimation method for a wide-field-of-view camera described in any one of claims 1 to 7 is implemented.

Images