Global target localization with multiple moving device cameras

Abstract

The invention relates to a positioning method for global positioning of a target (402) using a camera (103) of a mobile (portable) device (101) used by at least two operational personnel, without the need for a digital elevation map of the region.

Description

[0001] GLOBAL TARGET LOCALIZATION WITH MULTIPLE MOVING DEVICE CAMERAS

[0002] Technical Field

[0003] The invention relates to a positioning method that enables global positioning of a target without the need for a digital elevation map of the region using a portable device camera used by at least two operational personnel.

[0004] Prior Art

[0005] During the operation, soldiers are expected to first destroy the enemy elements they encounter with their own weapons, and if this is not sufficient, to accurately locate the enemy and report the location of the enemy to the support weapons (artillery, mortars, multi -barrel rocket launchers, etc.) or air elements (UCAVs, aircrafts, helicopters, etc.) in the back area. This is as true for the soldier serving in internal security operations as it is for the sentries / watchmen serving in border units.

[0006] Furthermore, DEM-dependent methods do not work for regions where DEMs are not accessible. In cases where GNSS data is inaccessible, global positioning of the target cannot be performed because the position of the mobile device is unknown.

[0007] Considering the existing target detection and positioning systems in the art, parameters such as the need for stabilization and calibration of these laser-based systems in the field, changes in measurement accuracy due to weather conditions, the need for re-installation during relocation, and warm-up and cooling times prevent them from being fast and ergonomic. Therefore, there is a need for handheld systems that are fast and easy to use, especially in tactical field activities, and that do not have related installation and handling issues. In addition, since the camera is a passive sensor, it has the ability to be concealed compared to laser-based sensors. Global positioning of the selected target using an inertial measurement unit (IMU), GNSS data and a digital elevation map (DEM) at the time of image acquisition is a well-known method. However, the resolution of the DEM is an important factor affecting the solution of the problem. A low resolution digital elevation map results in inaccurate positioning of the target selected from the image. At the same time, the digital elevation map is not obtained for every operational area. The large size of the digital elevation map creates a problem if the device has limited storage space. At the same time, preparing a digital elevation map of each different operational area brings problems in terms of accessibility and scalability.

[0008] In addition, the digital elevation map used in current applications also causes errors in terms of target positioning and the digital elevation map of the relevant region should be prepared and pre-loaded into the device.

[0009] The document titled Vision-based localization methods under GPS-denied conditions, in the state of the art, discusses the study of vision-based localization methods under GPS-denied conditions. The main flow is classified as Relative Vision Localization (RVL) and Absolute Vision Localization (AVL). The method is used in the area mapping domain. It is used for the positioning and area mapping of the object on which the camera is mounted, rather than any element in the camera's field of view. It proposes an algorithm for the localization of important elements in the image and their relative positions to the moving element to which the camera is attached.

[0010] Radar and Visual Odometry Integrated System Aided Navigation for UAVS in GNSS Denied Environment describes an integrated navigation system for Unmanned Aerial Vehicles (UAVs) in GNSS denied environments.

[0011] When the existing studies in the state of the art are examined, there was a need to develop a positioning method that enables the global positioning of the target without the need for a digital elevation map of the region with a portable device camera used by at least two operation personnel. Objectives of the Invention

[0012] The purpose of the present invention is to realize a positioning method that enables the global positioning of the target without the need for a digital elevation map of the region with a portable device camera used by at least two operation personnel.

[0013] Another object of the present invention is to realize a positioning method that does not require pre-installation and offers a very low cost solution compared to the currently used target range finders and target locators.

[0014] Another object of the present invention is to provide a positioning method that enables instant target positioning without the need for re-installation, even if the operation personnel change their position.

[0015] Another aim of the present invention is to realize a positioning method that improves the accuracy of user-dependent target selection, since the operational personnel can also see each other's marked targets instantaneously.

[0016] Detailed Description of the Invention

[0017] Exemplary configurations of the positioning method for achieving the objects of the present invention are shown in the attached figures.

[0018] These figures are;

[0019] Figure 1: A schematic view of an exemplary embodiment of the inventive method.

[0020] Figure 2: A graphical representation of the process of changing the beam origin used in the method of the invention.

[0021] Figure 3: Schematic view of Step 1 described in the stereo image solution used in the method of the invention.

[0022] Figure 4: Schematic view of Step 3 of the stereo image solution used in the method of the invention. The parts in the figures are individually numbered and the corresponding numbers are given below.

[0023] 101. Mobile device

[0024] 102. Inertial gauge I

[0025] 103. Camera

[0026] 104. Camera resolution

[0027] 105. Camera focal length

[0028] 106. Camera pixel size

[0029] 107. User

[0030] 108. Inertial gauge II

[0031] 201. Camera orientation

[0032] 202. Image

[0033] 203. Points of known location

[0034] 204. Blind navigation algorithm

[0035] 301. Pixel

[0036] 302. Pixel values

[0037] 400. Target location

[0038] 401. Target pixel

[0039] 402. Target

[0040] The invention relates to a positioning method which enables global positioning of a target (402) without the need for a digital elevation map of the region by means of a camera (103) of a portable device (101) used by at least two operation personnel; comprising the steps

[0041] Obtaining three-dimensional orientation information of mobile devices (101) with inertial gauge I (102) located on mobile devices (101),

[0042] Calculating roll, pitch and yaw values by converting the obtained orientation information into Euler angles through Kalman filter,

[0043] Capturing an image (202) of the target (402) area (402) by users (107) with the camera (103) of mobile devices (101), - Labeling of the captured mobile device (101) image (202) with the pitch, roll and roll angles obtained from the inertial meter I (102) on the mobile device (101),

[0044] Selecting points (203) with known positions on the captured image (202) and matching their positions with pixel values (302) on the image (202), Obtaining pixel values (302) in the image (202) captured with the camera resolution (104) of mobile devices (101),

[0045] - Finding the estimated location of the target point (402) from the obtained values, using the triangulation algorithm [3] and the recursive-iterative method,

[0046] Transforming the locations specified in the 3D coordinate plane into 2D focal planes using the modeling field theory (MFT) method [1],

[0047] Obtaining the orientation and position data of the users (107) by combining the data obtained from the inertial meter II (108) worn on the users (107) with a Kalman-based filter,

[0048] Calculating and averaging the acceleration values in three axes and the norm of the acceleration,

[0049] Calculating the maximum and minimum points by filtering the determined average acceleration values,

[0050] - Determining of the step detection after the calculation of the maximum and minimum values, if the minimum-maximum-minimum points are met consecutively,

[0051] Improving a route and creating a pedestrian blind navigation algorithm (204) by reusing the route data obtained from step detection, step length and orientation and the step detection calculated with the accelerometer, and by the classification of the user (107) activity obtained with the accelerometer,

[0052] - Determining of the target position (400) in the presence of an active route signal from the blind navigation algorithm (204).

[0053] Obtaining orientation information / Euler angles with IMU sensor

[0054] The mobile device (tablet PC, phone, etc.) to be used (101) has magnetometer, accelerometer and rotometer sensors, i.e. inertial meter I (102), which measure magnetic field, acceleration and angular velocity. Thanks to these sensors, three- dimensional camera orientation (201) information of the mobile device (101) can be obtained. The camera orientation (201) of the mobile device (101) is expressed in terms of the three angles used to describe the rotation of the device in three- dimensional space: yaw, pitch and roll, also known as Euler angles.

[0055] The raw data sent by the sensors built into the mobile device (101) will be in quadrature format, which is a four-dimensional complex number consisting of one real and three imaginary values. The quadrature format is a number system used to prevent the Gimbal Lock problem, which can occur when defining orientation with Euler angles, by coinciding with more than one angle with the same rotation matrix, i.e. losing degrees of freedom. Gimbal lock refers to a problem that occurs when using a three-axis motion sensor or IMU (Inertial Measurement Unit), especially when tracking the motion of an object represented using Euler angles. It refers to the situation where, when a platform is fixed in a certain position, the three Euler angles representing the motion of this platform are locked with each other. As a result, one degree of freedom is lost and it becomes difficult to track the motion of the object accurately.

[0056] The received data is passed through a Kalman filter for orientation estimation. The Kalman filter uses data from the rotometer for orientation estimation and data from the accelerometer and compass for correction. The filtered data are converted to Euler angles to obtain roll, pitch and roll values.

[0057] GNSS data calculation in non-GNSS environment

[0058] For position determination, firstly, an image (202) of the target (402) region is captured by a mobile device (101) camera (103) carried by the user (107). The captured mobile device (101) image (202) is labeled with the pitch, roll and roll angle measurements obtained from the inertial meter I (102) on the mobile device (101), and then points (203) with known positions on the captured image (202) are selected and their positions are matched with the pixel values (302) on the image (202). Finally, the roll, pitch and roll angles (camera orientation (201)), the field of view characteristics (104, 105, 106) of the camera (103) of the mobile device (101), and the pixel values (302) of the points with known locations (203) on the image (202) are provided as input to the method.

[0059] Pixel values (302) are obtained from the image (202) captured with the camera resolution (104) of the mobile device (101). The method is realized by marking the pixel values (302) and the points (203) whose location is known in advance on the camera (103) image (202) and matching them with the pixels (301). The location of the mobile device (101) is determined by obtaining the pixel values (302) from the pixels (301) in the image (202) and the points (203) whose location is known.

[0060] The algorithm used for location detection is based on the Modeling field theory[1]. The method given in Deming and Perlovsky[2]is modified and used to solve the location detection problem. In the Modeling field theory, a statistical model is created with the help of a data model including parameters such as location information, camera orientation (201) and sensor errors. The location information is estimated by maximizing the log-likelihood function, which gives a measure of how well the model fits the measured data set.[2]

[0061] Let the location of the camera (103) to be detected be A) = (A), Yj,Zf) j = 1 and the positions of K known points be xk= (xk, yk, zk) k = 1,2,3, The method first translates the positions of known points in the 3D coordinate plane xk= (xk,yk, zk), Xj = (A), Yj,Zj^ to the 2D focal plane according to the position of the camera (103) with the help of equation 1 and equation 2. In the equations, dy-is the focal length of the camera (105), d^md^ . are the elements of the direction orthogonality matrix, mdf elements are calculated using the roll

[0062] (co), pitch (K) and roll (cp) angles that give the camera orientation (201) as given in equation 3. Thus, the direction orthogonality matrix constructed using the elements in equation the last matrix can be expressed as right square brackets.

[0063] Based on Equations 1 and 2, being the target position (400) estimate, the coordinate points in terms of focus are found as The method corrects the estimates based on the similarity between the focal plane coordinate points collected as measurements and the coordinate points estimated by the system. This correction is done with the maximum likelihood estimator. In order to obtain the likelihood function, the distribution of measurements in the focal plane must first be calculated. This distribution can be calculated by equation 5. Using the law of total probability, the probability that a known point k comes from measurement j can be found as follows:

[0064] The likelihood function with the help of Equations 5 and 6 is expressed as

[0065] Due to the complex structure of the likelihood function in this case, the gradient descent method is preferred instead of taking the direct derivative and setting it equal to zero. The camera (103) position value to be improved is found iteratively by gradient descent as follows.

[0066] The step size in Equation 8 is chosen based on experience. Choosing a small step size results in convergence but requires a higher number of iterations.[2]Partial derivative in Equation 8 can be calculated with the set of equations Global positioning of target (402) with stereo image

[0067] A hybrid method for location finding in stereo image resolution is proposed that includes the triangulation method and the Modeling field theory (MFT)[1]. The triangulation method is basically based on two points with known locations and aims to find the third point with unknown location. The computational complexity of the triangulation method is quite low, but the problem of biased results arises as the noise on the angles made with the target (402) increases. In order to eliminate the bias problem, the triangulation method and the recursive-iterative method are used as a hybrid. The prerequisite for stereo image solution is that the target (402) to be selected for positioning is within the common field of view of the users' (107) cameras (103).

[0068] The steps of stereo image deconvolution for localization are described below.

[0069] Step 1 : The image (202) captured by any user (107) will be shared with other users (107) with the pixel location (u, v) of the selected target (402) on the image (202) and the location (X, Y, Z) of the user (107). u is the pixel value on the horizontal axis of the selected target (402) on the image (202), while v is the pixel value on the vertical axis. (Figure 3)

[0070] Step 2: After the target (402) is marked on the images (202), angle information ((|>, 0) is obtained from the marked pixels. The angle (|) is the angle calculated in the horizontal plane between the user (107) position and the target (402), while 9 is the angle calculated in the vertical plane.

[0071] Step 3: In this step, the triangulation algorithm [3] is applied using the angle information obtained. This algorithm uses the angle information between the camera (103) positions and the selected target (402) to find the estimated position of the target point (402). However, the solution found is biased (Figure 4).

[0072] Step 4: In this step, the Modeling field theory (MFT) method [1] is used to convert the locations specified in the 3D coordinate plane into a 2D focal plane. For each target (402), MFT corrects the predictions by looking at the similarity between the coordinate points in the focal plane collected as measurements and the coordinate points predicted and expected by the system. With this correction process, an unbiased solution is obtained.

[0073] In the stereo image solution, the target (402) to be selected for target (402) positioning should be within the field of view of each of the cameras (103) held by the users (107). The targets (402) in the image (202) will be positioned using the position information obtained by GNSS and the device orientation angles recorded at the time the photograph was taken.

[0074] Stereo image resolution does not use a digital elevation map as in ray tracing. Therefore, the performance of stereo image resolution will vary depending on the IMU and GNSS measurement accuracy. Errors due to the digital elevation map and performance limitations due to the resolution of the map will thus be eliminated.

[0075] User (107) improvement with blind navigation

[0076] The data obtained from the inertial meter II (108) mounted on the user (107) is combined with a Kalman-based filter to obtain the orientation and position data of the user (107). In addition, a route is generated by obtaining the step detection from the accelerometer data, the adaptive k-parameter step length, also obtained from the accelerometer data and calculated with the Weinberg model in Equation 16, and the device / step orientation (which is expected to be the same) obtained from the orientation.

[0077] In order to obtain the step detection, first the acceleration values in three axes and the norm of the acceleration are calculated. This is to better capture the variation in the vertical acceleration of human movement. From the norm value obtained, the average of the norm value is subtracted. The measurement accuracy of the inertial gauges (102, 108) and the point where they are placed on the pedestrian will cause noise in the sensor data. Filtering is performed to eliminate the effect of these noises. After filtering, the maximum and minimum points are calculated in the acceleration value. A threshold value is used to calculate the maximum and minimum points. Only the maximum and minimum points above a certain threshold value are kept. After calculating the maximum and minimum values, if the minimum-maximum-minimum points are met consecutively, the step is determined.

[0078] The route data obtained from step detection, step length and orientation, the reuse of the step detection calculated by the accelerometer and the classification of the user (107) activity obtained by the accelerometer are used to improve the route and create a pedestrian blind route.

[0079] As a result, if the user (107) receives an active route signal from the blind navigation algorithm (204), the precomputed target position (400) is updated.

[0080] After the target location (400) is determined, the user (107) selects a target pixel

[0081] (401) belonging to the target (402) whose location he / she wants to find as a result of the performed methods. The target pixel (401) is the selected pixel of the target

[0082] (402) in the image (202). This is the pixel that allows the location of the target (402) in the global coordinating system.

[0083] References:

[0084] 1. Perlovsky, Leonid I. "Neural networks and intellect: Using model-based concepts." (2001)

[0085] 2. Deming, Ross W., and Leonid I. Perlovsky. "Concurrent multi-target localization, data association, and navigation for a swarm of flying sensors." Information Fusion 8.3 (2007): 316-330.Deming ve Perlovsky (2007)

[0086] 3. Dogancay, Kutluyil, and Gokhan Ibal. "Instrumental variable estimator for 3D bearings-only emitter localization." 2005 International Conference on Intelligent Sensors, Sensor Networks and Information Processing. IEEE, (2005)

Claims

CLAIMS1. A positioning method which enables the global positioning of a target (402) without the need for a digital elevation map of the region with a camera (103) of a mobile (portable) device (101) used by at least two operation personnel; characterized by comprising the stepsObtaining three-dimensional orientation information of mobile devices (101) with inertial gauge I (102) located on mobile devices (101),- Calculating roll, pitch and yaw values by converting the obtained orientation information into Euler angles through Kalman filter,- Capturing an image (202) of the target (402) area (402) by users (107) with the camera (103) of mobile devices (101),- Labeling of the captured mobile device (101) image (202) with the pitch, roll and roll angles obtained from the inertial meter I (102) on the mobile device (101),Selecting points (203) with known positions on the captured image (202) and matching their positions with pixel values (302) on the image (202),- Obtaining pixel values (302) in the image (202) captured with the camera resolution (104) of mobile devices (101),- Finding the estimated location of the target point (402) from the obtained values, using the triangulation algorithm [3] and the recursive-iterative method,- Transforming the locations specified in the 3D coordinate plane into 2D focal planes using the modeling field theory (MFT) method [1],- Obtaining the orientation and position data of the users (107) by combining the data obtained from the inertial meter II (108) worn on the users (107) with a Kalman-based filter,- Calculating and averaging the acceleration values in three axes and the norm of the acceleration,Calculating the maximum and minimum points by filtering the determined average acceleration values,- Determining of the step detection after the calculation of the maximum and minimum values, if the minimum-maximum- minimum points are met consecutively,- Improving a route and creating a pedestrian blind navigation algorithm (204) by reusing the route data obtained from step detection, step length and orientation and the step detection calculated with the accelerometer, and by the classification of the user (107) activity obtained with the accelerometer,- Determining of the target position (400) in the presence of an active route signal from the blind navigation algorithm (204).

2. A positioning method according to claim 1, wherein the estimated position of the target point (402) is determined, characterized by comprising the stepsSharing the pixel location of the target (402) selected from the image (202) captured by any user (107) and with other users (107),- Obtaining the angle information of the target (402) for which the global position is to be found, from marked pixels after the target (402) is marked on the images (202),- Determining the estimated position of the target (402) point position using the triangulation algorithm [3],- Converting the locations first specified in the 3D coordinate plane to the 2D focal plane by the modeling field theory (MFT) method [1],3. A positioning method according to claim 1 , characterized the mobile device (101) being a tablet PC, a phone, etc.

Images