Target positioning with portable device camera in environments without global navigation satellite system (GNSS) data

Abstract

The invention relates to a positioning method for positioning the target in the image (202) in the global system by calculating the GNSS data with the information of the reference points within the field of view using a mobile (portable) device (101) camera (103) in environments where Global Navigation Satellite System (GNSS) information is not available, and calculating the position information changing depending on the movement of the mobile device (101).

Description

[0001] TARGET POSITIONING WITH PORTABLE DEVICE CAMERA IN ENVIRONMENTS WITHOUT GI XMIAL NAVIGATION SATELLITE SYSTEM (GNSS) DA A

[0002] Technical Field

[0003] The invention relates to a positioning method for positioning the target in the image in the global system by calculating the GNSS data with the information of the reference points within the field of view using a mobile (portable) device camera in environments where Global Navigation Satellite System (GNSS) information is not available, and calculating the position information changing depending on the movement of the mobile device.

[0004] Prior Art

[0005] Daring the operation, soldiers are expected to destroy the enemy elements they encounter primarily with their own weapons, and if this is not sufficient, to accurately locate the enemy and notify the support weapons (artillery, mortars, multi-barrel rocket launchers, etc.) or air elements (UCAVs, aircraft, helicopters, etc.) in the rear area. This situation is as valid for the guard / surveillance personnel serving in border units as it is for the soldier serving in internal security operations.

[0006] In the presence of GNSS, global positioning of the selected target using a DEM map is a known solution. However, in cases where GNSS data is not available, the global position of the mobile device is not known, so global positioning of the target cannot be performed.

[0007] Considering the existing target detection and positioning systems in the art, parameters such as the need for stabilization and calibration of these laser-based systems in the field, the change in measurement accuracy due to weather conditions, the need for re-installation during the change of location, and the warm-up and cooling times prevent these systems from being fast and ergonomic. Therefore, especially in tactical field operations, there is a need for hand-held systems that are fest and easy to use and do not have related installation and handling problems. In addition, since the camera is a passive sensor, it has the ability to be concealed compared to laser-based sensors.

[0008] Tekni in bilinen dutumunda yer alan Vision-based localization methods under GPS-denied conditions basiikh dokumanda, GPS'in kullamlraadtgt ortamlarda gdriis tabanh yerellestirme yontemlerinin incelenmesinden bahsedilmektedir. Ana akt§ Gbreceli Goru§ Konuralandirma CRVL) ve Mutlak Goriis Konumlandirma (AVL) olarak smiflandmlmaktadir. Yontem, alan haritalama (area mapping) alamnda kullamlmaktadir. Kamera gbrii§ a^ismdaki herhangi bir unsurun degil de kameram n uzerine takddigi unsurun konumlandinlmasr ve alanm harital ndmlmasi iyin kullamlmaktadir. Gbruntudeki onemli unsurl nn ve goreli konumlanmn hareket eden ve karaeramn bagb oldugu unsurun konumlandmlmasi i pin kollandan bir algoritma nermektedir.

[0009] In the document titled Vision-based localisation methods under GPS-denied conditions, discusses the study of vision-based localization methods in GPS- denied environments. The main flow is classified as Relative Vision Localization (RVL) and Absolute Vision Localization (AVL). The method is used in the field of area mapping. It is used for localization and area mapping of the object on which the camera is mounted, rather than any element in the camera's field of view. It proposes an algorithm for the localisation of important elements in the image and their relative positions to the moving element to which the camera is attached.

[0010] In the document titled Radar and Visual Odometry Integrated System Aided Navigation for LAVS in GNSS Denied Environment, an integrated navigation system developed for Unmanned Aerial Vehicles (UAVs) in GNSS denied environments is described.

[0011] When the existing studies in the prior art are examined, it is necessary to develop a positioning method that enables the target in the image to be positioned in the global system by calculating the GNSS data with the information of the reference points within the field of view using the portable device camera in environments where Global Navigation Satellite System (GNSS) information is not available, and calculating the position information changing depending on the movement of the portable device. Objectives of the Invention

[0012] The object of the present invention is to develop a positioning method that enables the target in the image to be positioned in the global system by calculating the GNSS data with the information of the reference points within the field of view using the portable device camera in environments where Global Navigation Satellite System (GNSS) information is not available, and calculating the changing position information depending on the movement of the portable device.

[0013] Another object of the present invention is to develop a positioning method that does not require pre-installation and offers a very low cost solution compared to the currently used target range finders and target locators. Another object of the present invention is to develop a positioning method that can obtain the coordinates of the target and report them instantly without recalibration, even if the operation personnel change their position in a non -GNSS environment.

[0014] Detailed Description of the Invention Exemplary embodiments of a positioning method for achieving the objects of the present invention are shown in the attached figures.

[0015] Figures,

[0016] Figure 1: A schematic view of an exemplary embodiment of the inventive method. Figure 2: A graphical view of the process of changing the beam starting point used in the inventive method. The pans in the figures are numbered individually, and the corresponding descriptions are given below

[0017] 101 . Mobile device

[0018] 102. Inertial measurement unit I

[0019] 103. Camera

[0020] 104. Camera resolution

[0021] 105. Camera focal length

[0022] 106 Camera pixel size

[0023] 107. Digital elevation model (DEM) map

[0024] 108. User

[0025] 109. Inertial measurement unit II

[0026] 201 . Camera orientation

[0027] 202 Image

[0028] 203. Points with known location

[0029] 204. Blind navigation algorithm

[0030] 301 . Pixel

[0031] 302. Pixel values

[0032] 400. Target position

[0033] 401. Target pixel

[0034] The invention relates to a positioning method for positioning a target with a camera ( 103) of a mobile device (101 ) in the absence of Global Navigation Satellite System (GNSS) data, it comprises;

[0035] Providing three-dimensional orientation information of the mobile device ( 101) with the inertial measurement unit I (102) located on the mobile device (101),

[0036] Calculating of roll, pitch and yaw values by converting the obtained orientation information into Euler angles through Kalman filter,

[0037] Capturing of the image (202) of the target area by the user (108) with the camera (103 ) of the mobile device ( 101 ), In the inventive method, the position of the user (108) of the mobile device ( 101) will be determined with the help of the points (three points) (203) whose position is known by the user (108) in cases where the GNSS is not working. For target detection, the position information of the mobile device (101 ) user (108), the camera orientation (201) (inertial measurement unit data) of the mobile device (101) and the image (202) from the camera (103) will be matched with the previously obtained digital elevation model (DEM) map (107)

[0038] The mobile device (tablet PC, phone, etc.) (101) to be used has magnetometer, accelerometer and rotometer sensors, i.e. inertial measurement unit 1 (102), which measure magnetic field, acceleration and angular velocity. Thanks to these sensors, three-dimensional camera orientation (201) information of the mobile device (101) can be obtained. The camera orientation (201) of the mobile device (101 ) is expressed in terms of the three angles used to define the rotation of the device in three-dimensional space, namely yaw, pitch and roll angles, also known as Euler angles.

[0039] The raw data sent by die sensors built into the mobile device (101 ) used will be in quadrature format, which is a four-dimensional complex number consisting of one real and three imaginary values. Quadrature format is a number system used to prevent the Gimbal Lock (Gimbal Lock) problem that may occur by coinciding with more than one angle with the same rotation matrix, i.e. losing degrees of freedom when defining orientation with Euler angles. Gimbal lock refers to a problem that occurs when using a three-axis motion sensor or IMU (Inertial Measurement Unit), especially when tracking the motion of an object represented using Euler angles It refers to the situation where, when a platform is fixed in a certain position, the three Euler angles representing the movement of this platform are locked with each other As a result, one degree of freedom is lost and it becomes difficult to accurately track the motion of the object.

[0040] The received data are passed through a Kalman filter for orientation estimation. The Kalman filter uses data from the rotometer for orientation estimation and data from the accelerometer and compass for correction The filtered data are converted to Euler angles to obtain roll, pitch and roll values.

[0041] For position determination, an image (202) of the target region is first captured by a mobile device (101) camera (103) carried by the user ( 108) The captured mobile device (101) image (202) is labelled with the yaw, pitch and roll angle measurements obtained from the inertial measurement unit I (102) on the mobile device (101), and then the points (203) with known positions on the captured image (202) are selected and the positions of these points are matched with the pixel values (302) on the image (202). Finally, the yaw, pitch and roll angles (camera orientation (201)), the field of view characteristics (104, 105, 106) of the camera (103 ) of the mobile device (101 ), and the pixel values (302) of the points with known positions (203) on the image (202) are provided as input to the method.

[0042] The pixel values (302) in the image (202) captured with the camera resolution (104) of the mobile device (101) are obtained. The method is realized by marking the pixel values (302) and the points (203) whose positions are known in advance on the camera (10.3) image (202) and matching them with the pixels (301). Mobile device (101) location detection is performed by obtaining pixel values (302) from the pixels (301 ) m the image (202) and the points (203) whose location is known.

[0043] The algorithm used in location detection is an algorithm based on the Field Modelling Theory!!J. The method given in Deming and Perlovsky is modified and used to solve the location detection problem. In the Field Modelling Theory, a statistical model is created with the help of a data model containing parameters such as position information, camera orientation (201 ) and sensor errors. The location information is estimated by maximizing the logdiketihood function, which gives a measure of how well the model fits the measured data set|2i

[0044] Assume that the location of the camera (103) to be detected is ) - wherein j::::l and the locations of K known points are wherein fc — 1,2,3, The method first converts the positions of the known points in the 3D coordinate plane into the 2D focal plane according to the position of the camera (103) at X. ~ ( I Z by means of equation 1 and equation 2.

[0045] . (x. -■ X,)mn-i- (y -- ly)mi2- (fo -■ ^>13

[0046] (fo ■--- YyJms! 4- (y / f- V)) 4- (z. ---- fo )m;: <

[0047] ('Equation 2)

[0048] In the equations, is the focal length of the camera (105), m - is the elements of the direction orthogonality matrix. elements are calculated using the yaw (©), pitch (K) and roll (cp) angles that give the camera orientation (201) as given in equation s. mn= cos<pcosK 12---■ cosars K 4- sinmfompcosx mJ3“ s u ruc ■--- cosms in^cosK 2!---- “Cosr sinK m-;.; — cosmcosK ---- s( ms( «psinK (Equation 3) m2 ;— SWWCOSK -i- cosmsin^sinK m35— stngi 32= -smmcos^ m33~ cosmcosg?

[0049] Thus, the direction orthogonality matrix formed using the elements in equation 3 can be expressed a mi2m. J

[0050] Based on Equations 1 and 2, ■■■ ( , Z, / ;) are the coordinate points found by the system in terms of focal as the target position (400) estimation as,

[0051] (Equation 4)

[0052] The method looks at the similarity between the focal plane coordinate points collected as measurements and the coordinate points estimated by the system and corrects the estimates. This correction process is performed with the maximum likelihood estimator. In order to obtain the likelihood function, the distribution of the measurements in terms of the focal plane must first be calculated. This distribution can be calculated by equation 5.

[0053] Using the law of total probability, the probability that the known point k comes from measurement / can be found as follows: in :: : P( Tfo Z' / fo)

[0054] (Equation 6)

[0055] The likelihood function with foe help of Equations 5 and 6 described as, (Equation 7)

[0056] Due to the complex structure of the likelihood function in this case, the gradient descent method was preferred instead of direct differentiation and equalization to zero. The camera (103) position value to be optimized is found iteratively by gradient descent as follows.

[0057] (Equation 8) The step size ,s in Equation 8 is chosen according to experience. Although choosing a small step size results in convergence, it requires a higher number of iterations. Partial derivative in Equation 8 calculated with a set of below equations.

[0058] In order to calculate the geographical coordinate of a point selected from an image (202) captured from a mobile device ( 101), various input parameters are required.

[0059] These parameters are as follows:

[0060] • Navigation data: This data usually consists of position (longitude, latitude, altitude) and IMU (yaw, roll, pitch) data.

[0061] • Digital elevation model (DEM) map (107): It is a map in raster format by using the height values taken from certain points of the earth's surface and defining the surface topography by interpolating the remaining surface with these values.

[0062] • Camera ( 103) parameters: Camera resolution ( 104), camera focal length ( 105), camera pixel size (106). In order to globally position the selected pixel (301 ) in the captured image (202), a ray tracingwbased positioning algorithm will be applied The steps to be applied in the algorithm are as follows: 1. A sensor model of the camera (103) is created by using the feature information (camera resolution (104), camera focal length (105), camera pixel size (106)) of the mobile device ( 101)

[0063] 2. Using the position information from the position sensor integrated in the mobile device (101) or from the position detection performed when the position sensor is not in operation, and the camera orientation (201) information obtained from the inertial measurement unit I (102) integrated in the mobile device (101), the position and angle of the rays extending from each camera (103) pixel (301) are calculated. The DEM map is used in this calculation process.

[0064] 3. In the process of calculating the position information of the selected pixel (301), the point where the rays sent from each camera (103) pixel (301) intersect the DEM map (107) is found.

[0065] 4. After calculating the distance of the found point to the camera (103), the selected point is translated using the position of the camera (103) and the angle of the camera (103 ).

[0066] 5. The translation vector is rotated with the camera (103) direction angle and the pixel values (302) of any selected pixel (301) are calculated

[0067] In the algorithm used for target positioning, the rays sent from the starting point with the ray tracing method can position the first intersection points in the map consisting of triangles. A ray R(t) with origin O and normalised direction D can be expressed as,

[0068] / ?(t) -• O 4- PD (Equation 10)

[0069] A triangle can consist of the vertices Point T(ri;v ) on the triangle is expressed as,

[0070] T(u, v) --- (1 -- u ---■ v)lft 4- ulft 4- ri(. (Equation 11) Here (u, v) are baricentric coordinates satisfying the conditions > 0, u > 0 and u + v < l. The intersection between the ray / (('ft and the point 77(u, v) is expressed by A(t) --- 7’(u, u). If we write this equation in detail, position data of the user. In addition, a route is created by obtaining the step detection obtained from the accelerometer data, the step length with adaptive k parameters, which is also obtained from the accelerometer data and calculated by the 'Weinberg model in Equation 16, and the device / step orientation obtained from the orientation (which is expected to be the same).

[0071] Step fenrfrh (Equation 16}

[0072] In order to obtain the step determination, the acceleration values in three axes and the norm of the acceleration are calculated first. The reason for this is to obtain the change in the vertical acceleration of human movement in a better way. From the norm value obtained, the average of the norm value is subtracted. The measurement accuracy of the inertial measurement units (102, 109} and the point where they are placed on the pedestrian will cause noise in the sensor data. Filtering is performed to eliminate the effect of these noises Maximum and minimum points are calculated in the acceleration value after the filtering process. A threshold value is used to calculate the maximum and minimum points Only the maximum and minimum points above a certain threshold value are kept. After the calculation of the maximum and minimum values, if the minimum-maximum- minimum points are achieved consecutively, the step is determined.

[0073] The route data obtained from step detection, step length and orientation, the reuse of the step detection calculated by the accelerometer and the classification of the user (108) activity obtained by the accelerometer are used to improve the route and create a pedestrian blind navigation.

[0074] As a result, if the user (108) receives an active route signal from the blind navigation algorithm (204), the pre-calculated target position (400) is updated

[0075] After the target position (400) is determined, the user selects a target pixel (401) of the target whose position is to be found as a result of the methods performed.

[0076] Target pixel (401) is the selected pixel of the target in the image. This is the pixel that allows the target to be located in the global coordinatization system.

Claims

Calculating and averaging the acceleration values in three axes and the norm of the acceleration,Calculating the maximum and minimum points by filtering the determined average acceleration values, ■■ Determining of the step identification after the calculation of the maximum and minimum values, if the minimum -maximumminimum points are provided consecuti ely.Creating of the pedestrian blind navigation algorithm (204) using the route data obtained from step detection, step length and orientation, the reuse of the step detection calculated with the accelerometer and the classification of the user (108) activity obtained with the accelerometer by improving the route. Determining of the target position (400) in the presence of an active route signal from the blind navigation algorithm (204).

2. A positioning method according to ciaim 1 , characterized by, ray tracingbased positioning algorithm comprises,- Creating of a sensor model of the camera (103) using the camera (103) feature information (camera resolution (104), camera focal length (105), camera pixel size (106)) of the mobile device (101), - Calculating the position and angle of the beams extending from each camera (103) pixel (301) using a DEM map (107) of the position information from the position sensor integrated in the mobile device (101 ) or from the position detection performed when the position sensor is not in operation and the camera orientation (201) information obtained from the inertial measurement unit I(102) integrated in the mobile device (101 ),- Calculating the location information of the selected pixel (301), the point where the rays sent from each camera (103) pixel (301) intersect the DEM map (107),

Images