Mapping method between latitude and longitude coordinates and image coordinates

Abstract

The present invention relates to a method for mapping longitude and latitude coordinates to image coordinates, which selects several points in a scene, uses known measurement data of the scene or tools such as Google Maps to calibrate the longitude and latitude coordinates in the physical space and the pixel coordinates in the image, establishes a corresponding relationship between the coordinate points in the image and the coordinates in the physical world, and on this basis, for any point in a given space, uses adjacent points of different numbers and sampling combinations to calculate a set of mapping points based on a homography matrix transformation model, and then deletes outliers with large deviations to obtain a final mapping result. This mapping method is not only highly accurate, but also conducive to reducing the amount of data processing.

Description

Technical Field

[0001] The invention relates to a method for mapping longitude and latitude coordinates to image coordinates. Background Art

[0002] In the enhanced display system, the relevant information of the objects in the scene needs to be enhanced and displayed at the corresponding position on the image; if the object of interest is moving, it is necessary to obtain the spatial position of the object in real time (such as the latitude and longitude coordinates from the Beidou system or GPS) and calculate its corresponding image coordinates. For example, in airport scene monitoring, in order to let the commander know the flight information and other related information of the moving aircraft on the video screen, it is necessary to receive the aircraft's own position information and identity information in real time, calculate the position of the aircraft in the picture based on the position information, and superimpose the identity information on it.

[0003] For moving targets on the ground, and when the camera remains stationary, there is a one-to-one correspondence between the longitude and latitude of the moving target and the pixels on the image. A common method to solve the coordinate mapping problem is to find the homography matrix [1] , at least four sets of corresponding points are needed to find the homography matrix.

[0004] For images acquired by a single-lens camera, the homography matrix strictly defines the mapping relationship between spatial latitude and longitude coordinates and image pixel coordinates. However, for images with a large field of view, such as panoramic images stitched from multiple lenses, it is impossible to express them in the form of a single analytical expression because the images undergo complex changes such as cropping, rotation, and shearing during the image stitching process. In addition, various geometric distortions (radial distortion, tangential distortion, etc.) in the imaging process are often difficult to completely correct in practice, which can also cause the homography matrix to fail.

[0005] In this case, the local transformation method can be used [2] For example, several feature points in the scene are selected in advance. These feature points divide the entire scene into multiple small local areas. After manually calibrating their corresponding image pixel coordinates, the mapping relationship between the spatial latitude and longitude coordinates and the image pixel coordinates can be established for each local area.

[0006] In the above local mapping, the calculation of homography matrix parameters is related to the selection of mapping point pairs. Since the manually calibrated coordinate values ​​contain errors in practical applications, and there is no guarantee that the calibration points used to calculate the homography matrix of each local area will simultaneously satisfy the single homography matrix constraints, the obtained image coordinates generally contain errors, and in some cases the errors are large. Summary of the invention

[0007] In order to overcome the above-mentioned defects of the prior art, the present invention provides a method for mapping longitude and latitude coordinates to image coordinates. This mapping method is not only highly accurate, but also helps to reduce the amount of data processing.

[0008] The technical solution of the present invention is: a method for mapping longitude and latitude coordinates to image coordinates, comprising the following steps:

[0009] 1) Establish calibration points:

[0010] Obtain the image coordinates of no less than 4 marker points (usually more than 4) in the image and the latitude and longitude coordinates in the real scene (physical space, or simply the scene), establish the corresponding relationship between the image coordinates and the latitude coordinates of these points, and use these points as calibration points;

[0011] 2) Select calibration points:

[0012] For any predicted point that is not a calibration point (the point whose latitude and longitude coordinates are to be mapped to image coordinates), find K calibration points in the vicinity of the predicted point based on the known latitude and longitude coordinates;

[0013] 3) Calculate the weighted homography matrix under each sampling method:

[0014] From the K calibration points in the selected neighboring area, n calibration points are extracted each time, 4≤n≤K, and at least three of the n calibration points are not collinear. Based on the n calibration points extracted each time, the weighted correspondence matrix H of the coordinate mapping is calculated according to the following method:

[0015]

[0016] make

[0017]

[0018]

[0019] h=(h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 ) T ,

[0020]

[0021] When n=4,

[0022] Solve Ah=B,

[0023] When n>4,

[0024] Solve h=(A T WA) -1 A T W.B.,

[0025] Solve for h,

[0026] Among them, w i is the weight coefficient of the ith calibration point. The closer the calibration point is to the point to be measured, the greater the weight coefficient is. When n=4, it can be considered that the weight coefficients of all calibration points are the same;

[0027] (x i ,y i ) is the image coordinate of the i-th calibration point among the n calibration points extracted,

[0028] (l i ,t i ) is the longitude and latitude coordinates of the ith calibration point,

[0029] i=1,2…n.

[0030] 4) Calculate the mapping coordinates under sampling:

[0031] For any point to be measured, according to the weighted homography matrix of the point to be measured under each sampling mode obtained in step 2) and the known longitude and latitude coordinates (l, t) of the predicted point, the following formula is used to calculate the estimated value of the image coordinates (x, y) under each sampling mode:

[0032]

[0033]

[0034] 5) Adaptive detection and mapping results:

[0035] The image point (x, y) set R obtained in step 3) is integrated according to the following method, including:

[0036] 5.1) Take the average of x and y of all points in set R;

[0037] 5.2) Traverse all points in R, find and delete the point farthest from the mean (x r ,y r ), when there are multiple points with the same distance from the mean (x r ,y r ), delete these farthest points together.

[0038] Repeat the above steps until the number of deleted points meets the set deletion requirement (including reaching the remaining point requirement), and take the average image coordinates of the remaining points in R as the image coordinates of the predicted point.

[0039] The scene may generally be a plane scene, where any position of the plane scene can be identified by longitude and latitude coordinates and the altitude is allowed to be ignored.

[0040] The image may be a panoramic image formed by stitching together a plurality of partial images.

[0041] The calibration points should usually be distributed over the entire scene involved in the image, and should not be distributed only in a certain area or be too concentrated in a certain area.

[0042] In step 2), a circle can be drawn with the predicted point as the center, and the calibration points falling within the circle are selected as calibration points in the vicinity of the predicted point. When the number of calibration points falling within the circle is too large or too small, the size of the circle can be adjusted and reselected.

[0043] In step 2), for any prediction point, after the number of calibration points n to be drawn each time is determined, sampling is performed in all possible sampling methods, or a certain number of random samplings are performed. When the number of samplings reaches the set number or the set conditions are met, no more sampling is performed.

[0044]

[0045] Among them, dist(l,t,l i ,t i ) is the Euclidean distance between the predicted point (l, t) and the i-th calibration point in the longitude and latitude coordinate system.

[0046] Based on the sampling calculation of homography matrix of known calibration points to realize the mapping of latitude and longitude coordinates and image coordinates, the present invention first finds the latitude and longitude coordinates of any point in the space, and first finds K (K≥4) calibrated points around the point, and then samples several points to calculate the homography matrix, and obtains the image coordinate mapping value under the sampling. Since there are many combinations of sampling, multiple samplings will obtain a set of coordinate mapping values, and then the influence of the values ​​with large deviations will be eliminated from this set of coordinate mapping values ​​to obtain the final image mapping coordinates.

[0047] Since the calibration points in the neighborhood of the predicted points are used as the basis for prediction, these calibration points have a great influence on the predicted points, which is beneficial to reducing unnecessary calculations while ensuring high prediction accuracy; thus, through multiple sampling methods, the number of calibration points based on each calculation is small, which helps to reduce the amount of calculation; since the weight coefficient is added on the basis of the homography matrix, the influence of the closer calibration points in the mapping is large, which is beneficial to improve the accuracy; since the mapping results under multiple sampling methods are integrated, it is helpful to reduce the impact of unreasonable mapping methods, and the rationality of the final mapping results is guaranteed by averaging the remaining mapping results. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described below in conjunction with the accompanying drawings.

[0050] The overall process of the present invention is as follows Figure 1 As shown. First, select several points in the scene, and use the known measurement data of the scene or tools such as Google Maps to calibrate their latitude and longitude coordinates in the physical space and their pixel coordinates in the image, that is, establish the correspondence between the coordinate points (x, y) in the image and the coordinates (l, t) in the physical world, expressed as several quadruples (x, y, l, t). Then, for any point in the given space, a set of mapping points is calculated based on the homography matrix transformation model using neighboring points of different numbers and sampling combinations. Then, the outliers with large deviations are deleted to obtain the final mapping result.

[0051] The main processes include:

[0052] 1. Coordinate calibration

[0053] Given an image, we first manually associate the image pixel coordinates with the physical world coordinates. Specifically, we look for obvious landmarks on the image (such as the corners or installation points of road signs, the corners of traffic lanes, the corners of ground decorations, etc.), record the coordinates (x, y), and then record the actual coordinates (l, t) of these points on a map that can display the physical world coordinates, recorded as (x, y, l, t).

[0054] The landmarks (calibration points) should cover the entire scene as much as possible and be distributed as evenly as possible.

[0055] 2. Calibration point sampling

[0056] There is a one-to-one correspondence between longitude and latitude coordinates and image coordinates, and the homography matrix is ​​used here to describe this correspondence. When the longitude and latitude of a point in space are known and its corresponding image coordinates are required, the homography matrix coefficients are first solved using the calibration points nearby, and then these coefficients are used to calculate the image coordinates corresponding to the point. In order to reduce the impact of calibration errors and fixed calibration point selection methods on the calculation of homography matrix coefficients, the calibration points are sampled in the present invention, and then each group of samples is used to calculate the homography matrix coefficients. Specifically, for a given point in space, find its nearest K calibration points, and select n (4≤n≤K) points from them, for a total of Then use these S combinations to solve the weighted homography matrix respectively.

[0057] 3. Weighted homography matrix mapping

[0058] The homography matrix has 8 degrees of freedom and is a 3×3 matrix in the following form:

[0059]

[0060] The transformation relationship between image coordinates and longitude and latitude is:

[0061]

[0062] Where s is an arbitrary scale, l and t are longitude and latitude, and x and y are the horizontal and vertical coordinates of the image plane space.

[0063] Expanding formula (2) yields formula (3):

[0064]

[0065] Write equation (3) in matrix form:

[0066]

[0067] Where h=(h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 ) T , the projection matrix can be determined by finding the 8 parameters in h.

[0068] The projection matrix between two planes (image and real scene) can be calculated by the selected n calibration points, but at least three points are not collinear. All the selected n calibration points are written into matrix form according to formula (4):

[0069]

[0070] Among them, the image coordinates of the i-th (i=1,2…n) point are (x i ,y i ), the longitude and latitude coordinates are (l i ,t i ). Substituting into the following equation, we can solve for h:

[0071] Ah=B Formula (6)

[0072] When n=4, equation (6) is well-posed and can be solved analytically; when n>4, equation (6) is over-determined and may only have an approximate solution.

[0073] Generally, the least squares method is used to unify these two situations:

[0074] Order error

[0075] E=(Ah-B) T (Ah-B) Formula (7)

[0076] When solving for the minimum value of E, the corresponding h is the required homography matrix.

[0077] In actual panoramic images, the number and accuracy of calibration points vary, and the distribution of calibration points near the predicted point (to be determined) is also difficult to estimate. For example, when estimating with four calibration points, there may be only three calibration points that are close to the predicted point, and the other calibration point is relatively far away. At this time, when predicting coordinates, it is obvious that the closer the calibration point, the greater its role should be, and a greater weight should be given.

[0078] Assume the weight of each calibration point is W:

[0079]

[0080] W is a diagonal matrix because the weight of each calibration point is only related to itself and has nothing to do with the distribution of other calibration points. i is the weight of each calibration point i,

[0081] Pick

[0082]

[0083] Among them, (l, t) is the latitude and longitude coordinates of the prediction point, (l i ,t i ) is the latitude and longitude coordinates of calibration point i (the i-th calibration point), and the dist function is used to calculate the Euclidean distance, dist(l,t,l i ,t i ) is the predicted point (l,t) and the calibration point (l i ,t i ) is the Euclidean distance between .

[0084] After considering the weights of different calibration points, formula (7) can be rewritten as:

[0085] E=(Ah-B) T W(Ah-B) Formula (10)

[0086] That is, the weight factor is added to the minimized error function, and during the optimization process, priority will be given to reducing the error of points with larger weights.

[0087] Take the partial derivative of h of E in equation (10) and set the derivative to 0 to solve:

[0088] h=(A T WA) -1 A T WB formula (11)

[0089] After obtaining h, substitute it into formula (3) to get the image coordinates of the prediction point.

[0090] 4. Adaptive prediction

[0091] Through the above steps, S mapping results are obtained, and the i-th (i=1,2…S) mapping result is recorded as (x ri ,y ri ), add these results to the set R,

[0092] R={(x r1 ,y r1 ),(x r2 ,y r2 ),…(x rS ,y rS )} Formula (12)

[0093] Each result in the set R can be regarded as an estimate of the final mapping result, but there may be errors. In order to reduce the error, the set R needs to be further integrated.

[0094] Take the average of all points x and y in the set R:

[0095]

[0096] Then traverse all the points in R and find the point farthest from the mean (x ri ,y ri ), and delete:

[0097] R←R-(x r ,y r ) Formula (14)

[0098] Repeat formula (13)-(14) until the number of deleted points reaches a certain proportion.

[0099] The choice of this ratio depends on the density of known calibration points and the degree of image distortion. Considering the convenience of implementation, 50% of R can usually be deleted, that is, S / 2 points (mapping results) can be deleted.

[0100] After deleting the points that deviate from the average value, the remaining points in R are relatively concentrated, and the average coordinate value of the remaining points is the image coordinate of the adaptive prediction.

[0101] The complexity of the mapping algorithm is related to the selected K. When K=4, the adaptive prediction process is omitted and the result of the weighted homography matrix is ​​directly returned. As K increases, the complexity of the algorithm increases, but the accuracy and stability of the prediction also increase. In practical applications, there are very few moving targets that need to calculate the mapping coordinates in real time, and the increase in algorithm complexity does not constitute a problem that requires special attention. In addition, the complexity of the algorithm can be controlled by changing the size of K and the number of selected combinations.

[0102] The longitude and latitude coordinates in the present invention can be equivalently replaced by other plane coordinates in the physical space.

[0103] Unless otherwise specified or one preferred or optional technical means is a further limitation of another technical means, all preferred and optional technical means disclosed in the present invention can be arbitrarily combined to form several different technical solutions.

[0104] References

[0105] [1] Qu Tianwei, An Bo. Homography matrix representation of two-dimensional projection transformation model[J]. Information Technology, 2008(03):88-90.

[0106] [2] Jiang Lu, Zheng Wentao, Wang Guofu. Aircraft placarding method based on video analysis and positioning information [P]. CN108133028A, 2018-06-08.

Claims

1. Method for mapping longitude and latitude coordinates to image coordinates, including the following steps: 1) Set up calibration points: Select several points in the scene, and use the known measurement data or tools of the scene to calibrate their longitude and latitude coordinates in the physical space and their pixel coordinates in the image frame, establish the correspondence between the coordinate points (x, y) in the image and the longitude and latitude coordinates (l, t), expressed as several quadruples (x, y, l, t), and use these points as calibration points; 2) Select calibration points: For any prediction point that is not a calibration point, according to the known longitude and latitude coordinates, find K calibration points in the adjacent area of this prediction point. For any prediction point, after the number n of calibration points extracted each time is determined, sample all possible sampling methods, or randomly sample a certain number of times. When the sampling times reach the set number or meet the set conditions, stop sampling; 3) Calculate the weighted homography matrix under sampling: Each time, extract n calibration points from the K calibration points in the selected adjacent area, 4 < n < K, and at least three of these n calibration points are not collinear. Based on the n calibration points obtained each time, calculate the weighted correspondence matrix H of coordinate mapping according to the following method: Let h=(h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 ) T , Solving h=(A T W.A. -1 A T WB, Solve for h, in, is the weight coefficient of the ith calibration point. The closer the calibration point is to the point to be measured, the greater the weight coefficient is. (x i ,y i ) is the image coordinate of the i-th calibration point among the n calibration points extracted, (l i ,t i ) is the longitude and latitude coordinates of the ith calibration point, i = 1, 2... n, dist(l,t,l i ,t i ) is the Euclidean distance between the predicted point (l, t) and the i-th calibration point in the longitude and latitude coordinate system; 4) Calculate the mapping results under each sampling method: For any point to be measured, use adjacent points with different numbers or different sampling combinations, and calculate a set of mapped points based on the homography matrix transformation model. According to the weighted homography matrix of this point to be measured under each sampling method obtained in step 3) and the known longitude and latitude coordinates (l, t) of this prediction point, use the following formula to calculate the estimated values of the image coordinates (x, y) under each sampling method respectively: 5) Adaptive detection and mapping coordinates: Integrate the set R of image points (x, y) obtained in step 4) according to the following method, including: 5.1) Take the average values of x and y of all points in the set R; 5.2) Traverse all points in R, find and delete the point farthest from the mean (x r ,y r ), Repeat the above steps until the number of deleted points meets the set deletion requirements, and use the average value of the image coordinates of the remaining points in R as the image coordinates of this prediction point; Control the algorithm complexity by changing the size of K and the number of selected combinations.

2. The method for mapping latitude and longitude coordinates to image coordinates as claimed in claim 1, characterized in that The scene is a planar scene, and the planar scene is a scene where any position can be identified by longitude and latitude coordinates and the height can be ignored.

3. The method for mapping latitude and longitude coordinates to image coordinates as claimed in claim 1, characterized in that The image is a panoramic image stitched by multiple local images.

4. The method for mapping latitude and longitude coordinates to image coordinates as claimed in claim 1, characterized in that The calibration points are distributed throughout the scene involved in the image.

5. The method for mapping latitude and longitude coordinates to image coordinates as claimed in claim 1, characterized in that In step 2), draw a circle with the prediction point as the center, and select the calibration points falling within this circle as the calibration points in the adjacent area of this prediction point. When the number of calibration points falling within this circle is too large or too small, adjust the size of the circle and re-select.

Images