Motion solver apparatus and method thereof

Abstract

A motion solver apparatus for estimating a relative planar motion between two camera poses (pose i, pose j) is provided. The motion solver apparatus is operative to acquire a set of pair-wise corresponding scale- and orientation covariant features present in two images of a real-world scene, the two images captured with the two respective camera poses. The motion solver apparatus is further operative to, for each pair of the set of pair-wise corresponding features, compute the relative planar motion between the two camera poses independent of other ones of the set of pair-wise corresponding features. The motion solver apparatus is further operative to determine the estimated relative planar motion between the two camera poses as the relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of pair-wise corresponding features.

Description

[0001]MOTION SOLVER APPARATUS AND METHOD THEREOF Technical field The invention relates to a motion solver apparatus for estimating arelative planar motion between two camera poses, a method of estimating a relative planar motion between two camera poses, a corresponding computer program, a corresponding computer-readable data carrier, and a corresponding data carrier signal. Background Relative pose estimation is the process of determining the relativeorientation and translation of a camera from at least two images captured bythe camera. In general, there are six degrees of freedom in which a cameracan be positioned in 3D space, three translational components (x, y, z) and three rotational components (yaw, pitch, roll). In scenarios such as autonomous driving, where cameras are used,among other sensors, to build a model of the environment through which an autonomous vehicle such as a car or a robot is moving, the camera motion is often constrained, e.g., by fixating the camera on the car. Since the camerais mounted rigidly to the vehicle, the only rotation of the camera is caused bythe vehicle turning. In other words, the camera is moving in a (2D) plane, andthe motion is referred to as planar motion. Consequently, the degrees offreedom, or motion parameters, are reduced to three: the translation in z(forward / backward) and x (lateral), and an angle of rotation θ around the y-axis. Note that even though the road or other surface over which the vehicle is moving may be non-planar, e.g., due to hills and other types of curvature, the motion of the camera can locally be approximated by a planar motion, inparticular given the relatively small distances which autonomous vehiclesmove between capturing subsequent images. This is the case becauseautonomous driving applications typically require images to be captured rather frequently for algorithmic stability, but also to be able to detect other vehicles and pedestrians in time. Another example where planar motion is asuitable approximation of the real motion is aerial surveillance withUnmanned Aerial Vehicle (UAVs) which are travelling at a constant altitude.This is common in, e.g., agricultural applications and city surveillance.Furthermore, it is common practice to refine the final solution obtainedunder the assumption of planar motion, allowing for minor deviations from the planar motion model. When incorporated in a complete SimultaneousLocalization And Mapping (SLAM) framework, loop-closure detection andbundle adjustment typically compensate for such imperfections over time. A common approach for estimating the change in pose of a camerafrom two images captured by the camera is based on the following steps (akapipeline): 1. Detecting points of interest or keypoints in both images.2. Calculating visual descriptors for these keypoints.3. Matching the features (the tuples of keypoint and descriptor) byvisual appearance, using the visual descriptors, to obtain a set of tentativelypair-wise corresponding features in the two images.4. Estimating the relative movement of the camera between the twoimages using the tentative correspondences obtained from the previous step.Remaining outliers among the correspondences need to be pruned toimprove the motion estimation. In practice, this may be achieved using theRANSAC framework (RANdom SAmple Consensus). This results in a finalset of pair-wise corresponding features (the inliers) and an estimated relativepose, i.e., the motion of the camera between the two images.5. Optionally, refining the solution obtained in the previous set using thefinal set obtained in the previous step for the given motion model. For calibrated cameras, the relative pose of the camera can be estimated up to a similarity transformation. In the case of planar motion,which is illustrated in Fig. 1, this means that the absolute scale of thetranslation of the camera between two captured images, i.e., the translationbetween pose i and pose j (illustrated by a dashed arrow), remainsambiguous. In other words, it is not possible to determine the distancetravelled by the camera (the length of the dashed arrow between pose i andpose j in Fig. 1) from two images alone, only the direction of the translationcan be obtained. This missing piece of information can often be inferred from other sensors, e.g., odometers. Consequently, it is common practice to represent the translation of the camera as an angle φ, corresponding to the direction of the translation. In conclusion, the planar motion model can be represented using twodegrees of freedom, which can be expressed as two angles (also referred toas motion parameters): the direction of translation φ of the camera, and itsorientation θ (both relative to the camera pose at the starting point of thetranslation). In Fig. 1, the respective field-of-view of the camera is illustratedas triangles oriented along the z-axis, which in case of an autonomousvehicle is the centerline of the vehicle. The relative motion of a camera, i.e., the two angles φ and θ, betweentwo images can be estimated using, e.g., the approach described in “Fastand reliable minimal relative pose estimation under planar motion”, byS. Choi and J.-H. Kim, Image and Vision Computing, vol. 69, pages 103–112,Elsevier, 2018 (in the remainder referred to as “Choi and Kim”). The solverproposed by Choi and Kim requires two pair-wise correspondences between features in two images to estimate the motion parameters (the relative translation φ and the relative orientation θ). An alternative solution for estimating the relative motion of a camerauses affine features (“Minimal Solutions for Relative Pose With a SingleAffine Correspondence”, by B. Guan, J. Zhao, Z. Li, F. Sun, andF. Fraundorfer, 2020 IEEE / CVF Conference on Computer Vision and PatternRecognition (CVPR), pages 1926–1935, 2020,doi:10.1109 / CVPR42600.2020.00200, in the remainder referred to as “Guanet al.”). An affine correspondence consists of a pair-wise correspondencebetween features in two images in addition to an affine (linear) transformationwhich maps image points in the neighborhood of the feature in the first imageto image points in the neighborhood of the corresponding feature in thesecond image. That is, an affine correspondence not only represents thevisual similarity between the respective neighborhoods around two features,also referred to as point correspondence, but also how the neighborhooddeforms between the two images. The proposed solver only requires a singleaffine pair-wise correspondence between features in the two images. Thisreduces the number of required RANSC iterations and opens the possibilityto employ other methods than RANSAC, such as histogram voting, which iscomputationally about 10 to 100 times more efficient than RANSAC basedapproaches (see, e.g., “Performance evaluation of 1-point-RANSAC visualodometry”, by D. Scaramuzza, Journal of Field Robotics, vol. 28, pages 792–811, Wiley, 2011, doi: doi.org / 10.1002 / rob.20411). On the other hand,computing affine correspondences is computationally considerably moredemanding, therefore offsetting the gain in computational efficiency byemploying histogram voting or reducing the number of RANSAC iterations.For example, according to “ASIFT: An Algorithm for Fully Affine InvariantComparison” (by G. Yu and J.-M. Morel, Image Processing On Line, vol. 1,pages 11–38, 2011, doi:10.5201 / ipol.2011.my-asift), the complexity ofextracting ASIFT features is about 13.5 times that of extracting SIFTfeatures, and matching ASIFT features is about 180 times that of matchingSIFT features. Recently, D. Barath and Z. Kukelova demonstrated (“Relative Posefrom SIFT Features“, Computer Vision – ECCV 2022: 17th EuropeanConference, pages 454–469, 2022, doi:10.1007 / 978-3-031-19824-3_27, in the remainder referred to as “Barath and Kukelova”) that it is possible to interpret SIFT features as affine features. This is the case since SIFTfeatures contain information about scale and orientation. More specifically,the local affine transformation, which can be represented by a 2 × 2 lineartransformation ^, can be expressed as ^ = ^^^^^^ , where ^^ = ^^^^ for ^ = 1,2are Jacobians matrices, ^^ is a rotation matrix corresponding to the rotationof the SIFT feature within the i-th image, and ^^contains information aboutthe scale of the SIFT feature within the i-th image:^ ^^,^^^^ = ^ 0 ^^,^^ where ^^,^ and ^^,^ are the scales of the feature along the image axes ^and ^, respectively, and is the shear. From this observation, constraintsrelating the SIFT features and the affine transformation which they approximate are obtained, which in turn are used to derive novel constraints for epipolar geometry (two distinct camera poses viewing the same scene). Using SIFT features together with an approximation of the affine correspondence derived from the SIFT features allows using fewer point correspondences to estimate the complete motion model, compared to usingonly point correspondences.Summary It is an object of the invention to provide an improved alternative to the above techniques and prior art. More specifically, it is an object of the invention to provide improvedsolutions for estimating the relative pose of a camera, i.e., the change inpose of the camera between two camera poses, for planar motion, using twoimages captured by the camera with the respective poses. These and other objects of the invention are achieved by means of different aspects of the invention, as defined by the independent claims. Embodiments of the invention are characterized by the dependent claims. According to a first aspect of the invention, a motion solver apparatus for estimating a relative planar motion between two camera poses is provided. The motion solver apparatus comprises processing circuitry causing the motion solver apparatus to be operative to acquire a set of pair-wise corresponding scale- and orientation covariant features present in twoimages of a real-world scene. The two images are captured with the tworespective camera poses. The motion solver apparatus is further operativeto, for each pair of the set of pair-wise corresponding features, compute therelative planar motion between the two camera poses independent of other ones of the set of pair-wise corresponding features. The motion solver apparatus is further operative to determine the estimated relative planar motion between the two camera poses as the relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of pair-wise corresponding features. According to a second aspect of the invention, a method of estimating a relative planar motion between two camera poses is provided. The methodcomprises acquiring a set of pair-wise corresponding scale- and orientationcovariant features present in two images of a real-world scene. The two images are captured with the two respective camera poses. The methodfurther comprises, computing, for each pair of the set of pair-wisecorresponding features, the relative planar motion between the two camera poses independent of other ones of the set of pair-wise correspondingfeatures. The method further comprises determining the estimated relativeplanar motion between the two camera poses as the relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of pair-wise corresponding features.According to a third aspect of the invention, a computer program isprovided. The computer program comprises instructions which, when thecomputer program is executed by one or more processors comprised in a computing device, cause the computing device to carry out the methodaccording to an embodiment of the second aspect of the invention.According to a fourth aspect of the invention, a computer-readable datacarrier provided. The computer-readable data carrier has stored thereon thecomputer program according to the third aspect of the invention. According to a fifth aspect of the invention, a data carrier signal isprovided. The data carrier signal carries the computer program according tothe third aspect of the invention. Even though advantages of the invention have in some cases been described with reference to embodiments of the first aspect of the invention, corresponding reasoning applies to embodiments of other aspects of the invention. Further objectives of, features of, and advantages with, the invention will become apparent when studying the following detailed disclosure, thedrawings, and the appended claims. Those skilled in the art realize thatdifferent features of the invention can be combined to create embodimentsother than those described in the following. Brief description of the drawings The above, as well as additional objects, features and advantages of the invention, will be better understood through the following illustrative and non-limiting detailed description of embodiments of the invention, with reference to the appended drawings, in which: Fig. 1 schematically illustrates the geometry and parameters of planarmotion, in accordance with embodiments of the invention.Fig. 2 schematically illustrates a motion solver apparatus for estimatinga relative planar motion between two camera poses, in accordance with embodiments of the invention. Fig. 3 shows a flow chart illustrating a method of estimating a relativeplanar motion between two camera poses, in accordance with embodiments of the invention. Fig.4 illustrates histogram voting, in accordance with embodiments of the invention. Fig. 5 illustrates cumulative density functions of the angular error fortranslation and rotation, respectively, and processing time, for the motionsolver apparatus in accordance with embodiments of the invention incomparison with conventional motion solvers. All the figures are schematic, not necessarily to scale, and generally only show parts which are necessary in order to elucidate the invention, wherein other parts may be omitted or merely suggested. Detailed description The invention will now be described more fully herein after withreference to the accompanying drawings, in which certain embodiments ofthe invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. In the following, solutions for solving the planar motion model in epipolar geometry, i.e., estimating the relative planar motion of a camerabetween two camera poses (pose i and pose j illustrated in Fig. 1) based onimages of a real-world scene captured with the two respective camera poses,are described. Although Barath and Kukelova have demonstrated that using SIFT features together with an approximation of the affine correspondence derived from the SIFT features allows using fewer point correspondences to estimate the complete motion model, as compared to using only the pointcorrespondences, they have not addressed the situation of relative planarmotion. In the following, SIFT constraints for planar motion are derived using an explicit parameterization. It turns out that the planar motion problem can be reduced to a quadratic equation in one variable. As a result, the number ofpair-wise correspondences between features in the two images is reduced toa single correspondence. Different from the method proposed by Barath andKukelova, which are depending on RANSAC-based approaches for a robustestimation of the complete motion model, embodiments of the invention canutilize histogram voting instead of RANSAC, owing to the fact that only asingle pair-wise correspondence between features in the two images isrequired. In comparison with known solutions, the proposed solution is advantageous in that it is computationally less complex and thereforerelatively fast in terms of processing speed, with respect to both computingand matching features to obtain sets of tentatively corresponding features(i.e., detecting keypoints in the two images, calculating visual descriptors forthe detected keypoints, and pair-wise matching features by comparing their visual descriptors), as well as solving the motion model. More specifically, whereas the solution by Choi and Kim is relatively fast in computing andmatching features, it requires two pair-wise correspondences betweenfeatures in two images to estimate the motion parameters using a RANSACapproach, which is relatively slow in terms of processing speed. On the other hand, the solution by Guan et al. relies on computing and matching affinefeatures, which is computationally relatively slow, but uses the relatively fasthistogram voting to solve the planar motion model since only a singlecorrespondence between affine features is required. If implemented in autonomous vehicle applications, e.g., factory robots and the like, embodiments of the invention are advantageous in that they result in areduced power consumption and shorter response times.Assuming that the camera is moving in a plane, the coordinate systemcan be chosen such that ^ = 0, and any translation of the camera can beexpressed as ^ = [^^ 0 ^^]T. Further, the only rotation of the camera mayoccur around the y-axis, which can be expressed by a rotation matrix ^^.Since only the relative motion is to be determined, only the direction ^ of thetranslation ^ can be obtained, and the problem can be parametrized as: where the translation scale ^ is unknown and ^ is the relative orientation ofthe camera (aka the yaw angle) (see Fig. 1). Hence, the relative planarmotion has two degrees of freedom. The corresponding essential matrix between the views captured in the two images can be written as: ^= ^^[^]× (3),where [^]× is the matrix representation of the cross product of ^. Eq. (3) canbe expressed explicitly as: Fixing the scale, i.e., ^ = 1, Eq. (4) can be written as: to define the non-zero elements ^^, ^ = 1 … 4, of the essential matrix.The well-known rank constraint det(^) = 0 is trivially fulfilled, and theknown trace constraint 2^^T^ − tr(^^T)^ = 0 is reduced to a singleequation: ^^ ^ ^^ − ^^ − ^^ + ^^^ = 0 (6).Considering homogenous keypoints, a single linear equation in theelements ^^ (Eq. (5)) can be obtained by using the epipolar constraintbetween the two views. Given two corresponding homogenous points ^^ ↔^^, = [^^, ^^, 1]T for ^ = 1,2 are respective positions of thecorresponding keypoints in the two images, the linear equation is given by ^T^^^^ = 0. For relative planar motion, this can be reduced to: Barath and Kukelova have derived a new constraint for estimating thefundamental matrix which leverages properties of scale- and orientationcovariant features: ^^^^^^^ + ^^^^^^^ + ^^^^^^^ + ^^^^^^^ + ^^^^^ + ^^^^^ + ^^^^^^ + ^^^^^^ +^^^^^^ + ^^^^^^ + ^^^^ + ^^^^ = 0 (8). areorientations of the pair-wise corresponding features in the two images, ^ =^^ / ^^, where ^^and ^^are respective sizes of the features in the two images,and ^^ are the elements of the 3 × 3 fundamental matrix in row-major order.The new constraint of Eq. (8) relies on orientation and scale of thematched features, and can therefore not be used with orientation covariantfeatures that are not scale covariant, e.g., ORB features. By using the newconstraint in combination with the epipolar constraint for pointcorrespondences from Eq. (7), a second linear equation is added in theelements of the fundamental matrix for every pair of matched scale- andorientation covariant features. Hence, the number of pair-wisecorrespondences between scale- and orientation covariant features whichare necessary for solving the planar motion model is reduced from ^ to ⌈^ / 2⌉(where ⌈^⌉ denotes the ceiling of ^). For example, for the generalfundamental matrix, 7 point correspondences are needed, resulting in⌈^ / 2⌉ = 4 correspondences between scale- and orientation covariantfeatures. For the essential matrix, 5 point correspondences are needed,resulting in ⌈5 / 2⌉ = 3 correspondences between scale- and orientation covariant features. Finally, in the planar motion scenario, 2 pointcorrespondences are required, and therefore only a single correspondence(⌈2 / 2⌉ = 1) between scale- and orientation covariant features is needed tosolve the planar motion model. Since scale- and orientation covariant features contain informationabout scale and rotation, expressed by ^^, ^^, and ^, defined hereinbefore,they have a relation to the affine correspondence. These relations can be expressed as: ^^^^ + ^^^^ − ^^^ = 0 (9b), and^^^^ + ^^^^ − ^^^ = 0 (9c),where are the elements of ^ representing the local affine transformation: From the above equations, a new constraint for planar motion can bederived: which together with the known constraint for point correspondences in Eq. (7)results in two equations for each pair-wise correspondence between scale-and orientation covariant features. As is described further below, these twoequations can be cast into a quadratic equation in a single variable, whichcan be solved relatively efficiently.In the following, embodiments of a motion solver apparatus 210 forestimating a relative planar motion, also referred to as relative pose orchange in pose, between two camera poses are now described withreference to Fig. 2. The motion solver apparatus 210 may be a computingdevice, e.g., a control unit of an autonomous vehicle or a server, in particularan edge server. In particular, the motion solver apparatus 210 may becomprised in an autonomous vehicle 200 which further comprises acamera 220 arranged to capture images of real-word scenes while theautonomous vehicle is moving. In this case, the relative motion of the vehicle can be estimated based on two images of the environment which arecaptured by the camera 220. In order to estimate the absolute motion of thevehicle, in particular the distance travelled between pose i and pose j inFig. 1, additional sensor data, e.g., from an odometer, may be used.The motion solver apparatus 210 comprises processing circuitry 220which causes the motion solver apparatus 210 to become operative inaccordance with embodiments of the invention described herein. The processing circuitry 220 may comprise one or moreprocessors 221, such as Central Processing Units (CPUs), microprocessors,application processors, application-specific processors, Graphics Processing Units (GPUs), and Digital Signal Processors (DSPs) including imageprocessors, or a combination thereof, and a memory 222 comprising acomputer program 223, i.e., software, comprising instructions. Whenexecuted by the processor(s) 221, the instructions cause the motion solverapparatus 210 to be operative in accordance with embodiments of theinvention described herein. The memory 222 may, e.g., be a Random-Access Memory (RAM), a Read-Only Memory (ROM), a Flash memory, orthe like. The computer program 223 may be downloaded to the memory 222by means of a communications interface circuitry comprised in the motionsolver apparatus 210, as a data carrier signal carrying the computerprogram 223. The processing circuitry 220 may alternatively or additionallycomprise one or more Application-Specific Integrated Circuits (ASICs), Field- Programmable Gate Arrays (FPGAs), or the like, which are operative tocause the motion solver apparatus 210 to be operative in accordance withembodiments of the invention described herein. More specifically, the motion solver apparatus 210 becomes operativeto acquire a set of pair-wise corresponding scale- and orientation covariantfeatures present in two images of a real-world scene. The two images havebeen captured with the two respective camera poses, either by the samecamera 220 which moves in-between the two camera poses, or by twodifferent cameras having different poses. The first scenario is common inautonomous driving applications in which images captured by a camera 220,which is movable with the autonomous vehicle 200, are used for controllingthe autonomous vehicle. Typically, images are captured frequently so thatthe vehicle has only moved a short distance in-between two consecutiveimages, and a considerable number of details in the real-world scenes arecaptured in both images. The scale- and orientation covariant features may,e.g., be features which have been detected using the known Scale-InvariantFeature Transform (SIFT) computer-vision algorithm (see, e.g., D. G. Lowe,"Object recognition from local scale-invariant features", Proceedings of theInternational Conference on Computer Vision, vol. 2, pages 1150–1157,1999, doi:10.1109 / ICCV.1999.790410). Pair-wise correspondences betweensuch features, which may contain outliers, may be obtained by matching ofvisual descriptors calculated for these features, as is known in the art.The motion solver apparatus 210 is further operative to compute, foreach pair of the set of pair-wise corresponding features, the relative planar motion between the two camera poses independent of other ones of the setof pair-wise corresponding features. Preferably, the relative planar motionbetween the two camera poses is computed for each pair of the set of pair- wise corresponding features using a planar motion constraint. The planarmotion constraint encodes a relationship between pair-wise correspondingscale- and orientation covariant features and the relative pose problem underthe planar motion assumption. More specifically, if the relative planar motionbetween the two camera poses is represented by the direction ^ of therelative translation (aka the translation direction) and the relative orientation ^of the camera (the change in orientation of the field-of-view of thecamera 220), the planar motion constraint can be expressed as in Eq. (11).Advantageously, the respective orientations ^^ and ^^ of the pair-wisecorresponding features in the two images, and the respective sizes ^^ and ^^of the features in the two images, or their ratio ^, approximate the affinetransformation between the two corresponding features. The motion solver apparatus 210 may further be operative to computethe relative planar motion between the two camera poses for each pair of the set of pair-wise corresponding features further using an epipolar constraintfor homogenous points. This constraint is commonly known as the “epipolarconstraint” and applies to homogeneous points, i.e., does not considerorientation nor scale of corresponding features. More specifically, if therelative planar motion between the two camera poses is represented by thedirection ^ of the relative translation and the relative orientation ^, asdescribed above, the epipolar constraint for homogenous points can beexpressed as in Eq. (7).The motion solver apparatus 210 may further be operative to computethe relative planar motion between the two camera poses for each pair of the set of pair-wise corresponding features by solving a quadratic equationderived from the planar motion constraint (Eq. (11)), the epipolar constraintfor homogenous points (Eq. (7)), and the trace constraint (Eq. (6)).In order to derive the quadratic equation, Eqs. (7) and (11) can beexpressed as a system ^^ = 0: The nullspace of ^ can be computed analytically, using singular valuedecomposition, or using QR decomposition. If ^ denotes a matrix of size4 × 2 which spans the nullspace denoting the elements of ^,the quadratic equation can be written, with ^ = ^^^ + ^^^ and fixing thescale by setting ^ = 1, as: Expanding this expression results in: where: ^^ = ^^^^^^ − ^^^^^^ − ^^^^^^ + ^^^^^^ (15b), and^^ = ^^^^ − ^^^^ − ^^^^ + ^^^^ (15c), which can be solved using the standard formula for finding the roots of aquadratic equation. For its two roots ^^ and ^^, the two solutions for thevector ^ are:^(^) = ^^^^ + ^^ (16a) and where ^^ and ^^ are the first and second column of ^, respectively. Fromthis, two possible solutions for the motion parameters, i.e., two pairs (^^, ^^)and (^^, ^^), corresponding to ^(^) and ^(^), respectively, can be obtainedthrough the following trigonometric identities: for ^ = 1,2, where arctan2(^, ^) is the angle between the x-axis and a rayfrom the origin to a point (^, ^) in the Cartesian plane.The motion solver apparatus 210 is further operative to determine theestimated relative planar motion between the two camera poses as the relative planar motion which statistically best represents the relative planarmotions computed for each pair of the set of pair-wise corresponding features. Preferably, the motion solver apparatus may be operative todetermine the relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of pair-wise corresponding features as one of: a most frequent range, an average, and amedian, of the set of relative planar motions computed for each pair of the set of pair-wise corresponding features. In practice, in order to determine the relative planar motion whichstatistically best represents the relative planar motions as a most frequent range of the set of relative planar motions computed for each pair of the setof pair-wise corresponding features, the set of computed planar motions maybe analyzed using histogram voting, as is illustrated in Fig. 4. Morespecifically, the computed values for the direction φ of the relative translationand the relative orientation θ are distributed in bins of histograms, onehistogram for each parameter, of typically equal range. For each histogram,the bin with most values is determined. This bin, or rather a valuerepresenting the bin, e.g., the center value of the bin, is the value which statistically best represents the relative planar motion of the camera. In thisway, the values for φ and θ which statistically best represent the relativeplanar motion of the camera are obtained. As an alternative, the relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of pair-wise corresponding features may be determined by calculating medianvalues, one for the direction φ of the relative translation and one for therelative orientation θ. Calculating median values is typically computationallyless complex than histogram voting while at the same time providing a goodapproximation, as is illustrated in Fig. 4.The motion solver apparatus 210 may further be operative to iterativelyremove outliers from the set of pair-wise corresponding features, and determine the estimated relative planar motion between the two camera poses as the relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of remaining pair- wise corresponding features, i.e., the set of inliers. The removed outliers arepair-wise corresponding features with respective computed relative planarmotions which deviate from the determined relative planar motion whichstatistically best represents the relative planar motions computed for each pair of the set of pair-wise corresponding features. Advantageously, by removing outliers from the set of pair-wise corresponding features, the planar motion between the two camera poses may be estimated more accurately.The motion solver apparatus 210 may further be operative to acquirethe set of pair-wise corresponding scale- and orientation covariant featurespresent in the two images of a real-world scene by detecting scale- andorientation covariant keypoints in the two images, calculating visualdescriptors for the detected keypoints, and matching the calculated visualdescriptors to obtain the set of pair-wise corresponding scale- and orientationcovariant features. The scale- and orientation covariant features may bedetected using a known computer vision algorithm for detecting local featuressuch as SIFT, SURF, LFNet, HesAffNet, CovDet, or S3Esti.The solution proposed herein has been tested using the well-knownKITTI dataset odometry (A. Geiger, P. Lenz, and R. Urtasun, “Are we readyfor Autonomous Driving? The KITTI Vision Benchmark Suite”, 2012 IEEEConference on Computer Vision and Pattern Recognition, pages 3354–3361,IEEE, 2012, doi:10.1109 / CVPR.2012.6248074), which consists of 11 image sequences of a car navigating urban environments, annotated with their ground-truth poses. In order to compare the performance of the proposed solution with known solutions, the following metrics have been used: -The translation error ^^ as defined by Choi and Kim.- The rotation error ^^ as defined by Choi and Kim.- The percentage of inliers detected.- The processing time of the solver.The below table summarizes the performance of three differentmethods using the GC-RANSAC framework (D. Barath and J. Matas, “Graph-Cut RANSAC”, 2018 IEEE / CVF Conference on Computer Vision and PatternRecognition, pages 6733–6741, IEEE, 2018, doi:10.1109 / CVPR.2018.00704)with sequence 04 of the KITTI dataset. GC-RANSAC is a state-of-the-artRANSAC-type framework which applies non-minimal refinement to the best solution iteratively, which improves performance (but also impacts computational complexity). Furthermore, it optimizes how to select inliers in amore sophisticated way than only selecting inliers uniformly at random. Thetable shows the median performance for the whole sequence of images.Number of Inliers Processing Method^ [deg] ^ [deg]correspondences^ ^[%] time [ms] Choi and 2 point 0.8959 0.3509 80.79 10.83Kim correspondences Guan et 1 affine 0.5838 0.2409 81.87 15.55al. correspondence 1scale- andCurrent orientation 0.5692 0.2717 84.25 8.57proposal covariant correspondence As can be seen, the solution proposed herein is relatively fast andperforms best for most metrics.The performance of the proposed solution can further be assessed byconsidering the cumulative density functions for the angular error in translation and rotation, respectively, as well as for the processing time.These are illustrated in Fig. 5 for sequence 04 of the KITTI dataset, using theGC-RANSAC framework. A method is considered accurate (for angular error)or fast (for processing time) if its curve is on the left side of the diagram.While all methods demonstrate to achieve similar accuracy, the proposedmethod is relatively faster. For example, the proposed method has an 80% ofprobability of achieving the solution in approximately 8 ms, while othersrequire at least approximately 10 ms.In the following, embodiments of a method 300 of estimating a relativeplanar motion between two camera poses (pose i and pose j in Fig. 1) aredescribed with reference to Fig. 3. Embodiments of the method 300 may, e.g., be performed by a computing device such as an autonomous vehicle, a robot, a server, a motion solver apparatus, or the like. The method 300 comprises acquiring 301 a set of pair-wisecorresponding scale- and orientation covariant features present in twoimages of a real-world scene. The two images are captured with the tworespective camera poses. The method 300 further comprises computing 302,for each pair of the set of pair-wise corresponding features, the relative planar motion between the two camera poses independent of other ones ofthe set of pair-wise corresponding features. The method 300 furthercomprises determining 303 the estimated relative planar motion between thetwo camera poses as the relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of pair-wise corresponding features. The relative planar motion between the two camera poses may becomputed 302, for each pair of the set of pair-wise corresponding features,using a planar motion constraint which encodes a relationship between pair-wise corresponding scale- and orientation covariant features and the relativepose problem under the planar motion assumption. Preferably, if the relativeplanar motion between the two camera poses is represented by a direction ^of a relative translation and a relative orientation ^, the planar motionconstraint can be expressed as in Eq. (11).The relative planar motion between the two camera poses mayoptionally be computed 302, for each pair of the set of pair-wisecorresponding features, further using an epipolar constraint for homogenouspoints as in Eq. (7).The relative planar motion between the two camera poses mayoptionally be computed, for each pair of the set of pair-wise correspondingfeatures, by solving a quadratic equation derived from the planar motionconstraint (Eq. (11)), the epipolar constraint for homogenous points (Eq. (7),and a trace constraint as in Eq. (6).The method 300 may further comprise removing 304 outliers from theset of pair-wise corresponding features, and determining 303 the estimatedrelative planar motion between the two camera poses as the relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of remaining pair-wise correspondingfeatures. The outliers may be removed by removing pair-wise correspondingfeatures with respective computed relative planar motions which deviate fromthe determined relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of pair-wise corresponding features. The relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of pair-wise correspondingfeatures may be determined 303 as one of: a most frequent range, anaverage, and a median, of the set of relative planar motions computed foreach pair of the set of pair-wise corresponding features.The acquiring 301 the set of pair-wise corresponding scale- andorientation covariant features present in the two images of a real-world scenemay comprise detecting scale- and orientation covariant keypoints in the twoimages using a computer vision algorithm for detecting local features,calculating visual descriptors for the detected keypoints, and matching thecalculated visual descriptors to obtain the set of pair-wise correspondingscale- and orientation covariant features.It will be appreciated that the method 300 may comprise additional,alternative, or modified, steps in accordance with what is describedthroughout this disclosure. An embodiment of the method 300 may beimplemented as the computer program 223 comprising instructions which,when the computer program 223 is executed by one or moreprocessor(s) 221 comprised in a computing device 210, such as a motionsolver apparatus, cause the computing device 210 to carry out themethod 300 and become operative in accordance with embodiments of theinvention described herein. The computer program 223 may be stored in acomputer-readable data carrier, such as the memory 222. Alternatively, thecomputer program 223 may be carried by a data carrier signal, e.g.,downloaded to the memory 222 via a communications interface circuitrycomprised in the computing device 210.The person skilled in the art realizes that the invention by no means is limited to the embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims.

Claims

CLAIMS 1. A motion solver apparatus (210) for estimating a relative planarmotion (^, ^) between two camera poses (pose i, pose j), the motion solverapparatus comprising processing circuitry causing the motion solverapparatus to be operative to: acquire a set of pair-wise corresponding scale- and orientationcovariant features present in two images of a real-world scene, the twoimages captured with the two respective camera poses,for each pair of the set of pair-wise corresponding features, compute the relative planar motion between the two camera poses independent of other ones of the set of pair-wise corresponding features, and determine the estimated relative planar motion between the two camera poses as the relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of pair-wise corresponding features.

2. The motion solver apparatus (210) according to claim 1, operative tocompute the relative planar motion between the two camera poses for each pair of the set of pair-wise corresponding features using a planar motion constraint encoding a relationship between pair-wise corresponding scale-and orientation covariant features and the relative pose problem under theplanar motion assumption.

3. The motion solver apparatus (210) according to claim 2, wherein therelative planar motion between the two camera poses is represented by adirection ^ of a relative translation and a relative orientation ^, and the planarmotion constraint can be expressed as:where [^^, ^^, 1] and [^^, ^^, 1] are respective positions of the pair-wisecorresponding features in the two images, ^^ = cos ^^= sin ^^, where^^and ^^are respective orientations of the pair-wise corresponding featuresin the two images, ^ = ^^ / ^^, where ^^ and ^^ are respective sizes of thefeatures in the two images, ^^ = cos(^ − ^), ^^ = − cos ^, ^^ = sin ^, and^^ = sin(^ − ^).

4. The motion solver apparatus (210) according to claim 2 or 3,operative to compute the relative planar motion between the two camera poses for each pair of the set of pair-wise corresponding features further using an epipolar constraint for homogenous points.

5. The motion solver apparatus (210) according to claim 4, wherein therelative planar motion between the two camera poses is represented by adirection ^ of a relative translation and a relative orientation ^, and theepipolar constraint for homogenous points can be expressed as: ^^^^^^ + ^^^^^^ + ^^^^ + ^^^^ = 0,where [^^, ^^, 1] and [^^, ^^, 1] are respective positions of the pair-wisecorresponding features in the two images, ^^ = cos(^ − ^), ^^ = − cos ^, ^^ =sin ^, and ^^ = sin(^ − ^).

6. The motion solver apparatus (210) according to claim 4 or 5,operative to compute the relative planar motion between the two camera poses for each pair of the set of pair-wise corresponding features by solvinga quadratic equation derived from the planar motion constraint, the epipolarconstraint for homogenous points, and a trace constraint ^^^ − ^^^ − ^^^ + ^^^= 0.

7. The motion solver apparatus (210) according to any one of claims 1to 6, further operative to iteratively:remove outliers from the set of pair-wise corresponding features withrespective computed relative planar motions which deviate from thedetermined relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of pair-wise corresponding features, wherein the estimated relative planar motion between the two camera poses is determined as the relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of remaining pair-wise corresponding features.

8. The motion solver apparatus (210) according to any one of claims 1to 7, operative to determine the relative planar motion which statistically bestrepresents the relative planar motions computed for each pair of the set of pair-wise corresponding features as one of: a most frequent range, an average, and a median, of the set of relative planar motions computed foreach pair of the set of pair-wise corresponding features.

9. The motion solver apparatus (210) according to any one of claims 1to 8, operative to acquire the set of pair-wise corresponding scale- andorientation covariant features present in the two images of a real-world sceneby: detecting scale- and orientation covariant keypoints in the two imagesusing a computer vision algorithm for detecting local features, calculating visual descriptors for the detected keypoints, andmatching the calculated visual descriptors to obtain the set of pair-wisecorresponding scale- and orientation covariant features.

10. An autonomous vehicle (200) comprising:the motion solver apparatus (210) according to any one of claims 1 to 9,anda camera (220) arranged to capture images of real-word scenes whilethe autonomous vehicle is moving.

11. A method (300) of estimating a relative planar motion between twocamera poses (pose i, pose j), the method comprising:acquiring (301) a set of pair-wise corresponding scale- and orientationcovariant features present in two images of a real-world scene, the two images captured with the two respective camera poses, for each pair of the set of pair-wise corresponding features,computing (302) the relative planar motion between the two camera posesindependent of other ones of the set of pair-wise corresponding features, and determining (303) the estimated relative planar motion between the twocamera poses as the relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of pair-wise corresponding features.

12. The method (300) according to claim 11, wherein the relative planarmotion between the two camera poses is computed (302) for each pair of theset of pair-wise corresponding features using a planar motion constraintencoding a relationship between pair-wise corresponding scale- andorientation covariant features and the relative pose problem under the planarmotion assumption.

13. The method (300) according to claim 12, wherein the relative planarmotion between the two camera poses is represented by a direction ^ of arelative translation and a relative orientation ^, and the planar motionconstraint can be expressed as:where [^^, ^^, 1] and [^^, ^^, 1] are respective positions of the pair-wisecorresponding features in the two images, ^^ = cos ^^= sin ^^, where^^and ^^are respective orientations of the pair-wise corresponding featuresin the two images, ^ = ^^ / ^^, where ^^ and ^^ are respective sizes of thefeatures in the two images, ^^ = cos(^ − ^), ^^ = − cos ^, ^^ = sin ^, and^^ = sin(^ − ^).

14. The method (300) according to claim 12 or 13, wherein the relativeplanar motion between the two camera poses is computed (302) for eachpair of the set of pair-wise corresponding features further using an epipolar constraint for homogenous points.

15. The method (300) according to claim 14, wherein the relative planarmotion between the two camera poses is represented by a direction ^ of arelative translation and a relative orientation ^, and the epipolar constraint forhomogenous points can be expressed as: ^^^^^^ + ^^^^^^ + ^^^^ + ^^^^ = 0,where [^^, ^^, 1] and [^^, ^^, 1] are respective positions of the pair-wisecorresponding features in the two images, ^^ = cos(^ − ^), ^^ = − cos ^, ^^ =sin ^, and ^^ = sin(^ − ^).

16. The method (300) according to claim 14 or 15, wherein the relativeplanar motion between the two camera poses is computed (302) for eachpair of the set of pair-wise corresponding features by solving a quadraticequation derived from the planar motion constraint, the epipolar constraint forhomogenous points, and a trace constraint ^^ ^ ^ ^^ − ^^ − ^^ + ^^ = 0.

17. The method (300) according to any one of claims 11 to 16, furthercomprising iteratively:removing (304) outliers from the set of pair-wise corresponding featureswith respective computed relative planar motions which deviate from thedetermined relative planar motion which statistically best represents therelative planar motions computed for each pair of the set of pair-wise corresponding features, wherein the estimated relative planar motion between the two camera poses is determined as the relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of remaining pair-wise corresponding features.

18. The method (300) according to any one of claims 11 to 17, whereinthe relative planar motion which statistically best represents the relative planar motions computed for each pair of the set of pair-wise correspondingfeatures is determined (303) as one of: a most frequent range, an average,and a median, of the set of relative planar motions computed for each pair of the set of pair-wise corresponding features.

19. The method (300) according to any one of claims 11 to 18, whereinthe acquiring (301) the set of pair-wise corresponding scale- and orientationcovariant features present in the two images of a real-world scenecomprises: detecting scale- and orientation covariant keypoints in the two imagesusing a computer vision algorithm for detecting local features, calculating visual descriptors for the detected keypoints, and matching the calculated visual descriptors to obtain the set of pair-wisecorresponding scale- and orientation covariant features.

20. A computer program (223) comprising instructions which, when thecomputer program (223) is executed by one or more processors (221)comprised in a computing device (210), cause the computing device (210) tocarry out the method (300) according to any one of claims 11 to 19.

21. A computer-readable data carrier (222) having stored thereon the computer program (223) according to claim 20.

22. A data carrier signal carrying the computer program (223) according to claim 20.

Images