A method for determining geographical position

Abstract

Using geographical coordinates and geographical information described by map data, a determination is made of map geographical features that are within the line of sight of the geographical coordinate The determined map geographical features are represented with a respective map value. A digital image is obtained, the image being recorded at a first geographical position of a landscape visible at the first geographical position. Image geographical features are identified in the digital image and a query data structure is created. The query data structure comprises image values that represent the identified image geographical features in the digital image. A matching is then performed of the query data structure with 0 the map values representing the map geographical features. An estimate is thereby obtained of a geographical coordinate for the first geographical position at which the digital image was recorded.

Description

[0001] A METHOD FOR DETERMINING GEOGRAPHICAL POSITION

[0002] TECHNICAL FIELD

[0003] The present invention relates to geographical positioning by using information on geographical features from map data and information based on a digital image.

[0004] BACKGROUND

[0005] Many positioning systems rely on external signal sources such as radio transmissions from mobile base stations, radar and / or sonar reflections, or global navigation satellite system (GNSS) signals. These systems are typically vulnerable to disruptions of the signal sources, or in that the transmitted signals reveal the position of the positioning system. Other positioning systems include image aided positioning, or terrain aided positioning, which relies on unique terrain or elevation features to find correspondences, as opposed to visual features. Terrain Contour Matching (TERCOM) is an example of a terrain aided positioning system, which often uses radar altimeter measurements to compute a history of terrain heights and correlates that history with terrain heights from a database map. Such systems also suffer from revealing the position of the positioning system to an external observer. For instance, it is relatively easy to accurately localize the origin of a radar or sonar signal. Thus, having a positioning system that do not rely on such external signal sources that may be easily disrupted or on techniques that reveals the position of the positioning system is highly sought after.

[0006] SUMMARY

[0007] With the above in mind, an object of the present invention is to provide a positioning system, which seeks to mitigate, alleviate or eliminate one or more of the above-identified deficiencies in the art and disadvantages singly or in any combination. Hence, in a first aspect of the present invention there is provided a method, which may be performed by processing circuitry, of determining a geographical position. The method comprises determining, for at least one geographical coordinate from a set of map data and using geographical information described by the map data, map geographical features that are within the line of sight of the geographical coordinate. Geographical features may be features such as water, land, land edge, distance to land edge, height of land, fixed objects such as buoys and buildings.

[0008] The determined map geographical features are represented with a respective map value. A digital image is obtained, the image being recorded at a first geographical position of a landscape visible at the first geographical position. Image geographical features are identified in the digital image and a query data structure is created. The query data structure comprises image values that represent the identified image geographical features in the digital image. A matching is then performed of the query data structure with the map values representing the map geographical features. An estimate is thereby obtained of a geographical coordinate for the first geographical position at which the digital image was recorded.

[0009] Such a method is advantageous at least in that it enables determination of a geographical position while being independent on external signal sources such as radio transmissions from mobile base stations, radar and / or sonar reflections, or satellite-based positioning signals. Both map geographical features and image geographical features may be represented with values that exist in the same domain. This means that the amount of data and operations needed to perform matching of map data with visual features is greatly decreased (i.e. , such a matching procedure can be faster), as compared to e.g., computing intensive image recognition using the full digital image.

[0010] In various embodiments, the method may comprise creating a database containing a plurality of geographical coordinates associated with an area described by the map data and storing map values for respective geographical coordinate in the database. The method may further comprise providing the query data structure to the database in order to obtain an estimate of a matching map geographical feature in the database

[0011] In other words, a procedure of pre-processing may be performed where a database is created that contains geographical coordinates associated with an area described by the map data. Creation of such a database of map geographical features before-hand enables a significant improvement with regard to the processing speed of the matching process as compared to a matching process where the map geographical features are determined for every matching process. Fewer operations are required in the matching procedure when accessing the database as compared to determining, i.e., performing calculations, map geographical features for each matching procedure.

[0012] In various embodiments, the method may comprise arranging the map values and image values that represent map geographical features and image geographical features in vector-like data structures. By vector-like is meant any indexed data structure, such as vector, array, matrix or ordered set of any dimensionality. When a database is used for storing map values, this may also imply that the database is a vector-like database.

[0013] By arranging the values in vector-like data structures, it is possible to represent information on geographical features from both map data and digital image in the same domain. This is an advantage because it enables a significant improvement with regard to the processing speed of the matching process as compared to a matching process where the values are not arranged in vector-like data structures.

[0014] The map values may be arranged such that the position of each map value within the vector-like map data structure is representative of a respective heading with respect to said geographical coordinate at which the map geographical feature falls within the line of sight. A heading can either be only in the horizontal plane, or have two components, one in the horizontal plane and one in the vertical plane. When a heading interval henceforth is referenced, it is implied that the same reasoning holds for a corresponding heading area in the two-dimensional heading case.

[0015] The creation of the query data structure may comprise dividing the segmented digital image into a plurality of sectors, where each sector represents a sector heading interval of the digital image. The respective image values identified in each sector may be arranged such that the position of each image value within the query data structure is representative of the respective sector heading interval.

[0016] In other words, the values of the vector-like data structures may be arranged such that the index of each value is representative of a respective heading (with respect to the geographical coordinate at which the value originated) where the feature represented by the value is located. This eliminates the need to store information about heading. This may be advantageous because less data is needed as compared to storing the information about heading. This enables a faster matching process. An advantage of dividing the segmented digital image into sectors is that it enables storing the image value representative for each sector at one position in the query data structure, each position thus representing a sector heading interval.

[0017] In various embodiments, the method may comprise performing ray tracing along headings with respect to the geographical coordinate and detecting the presence of map geographical features along each ray traced heading.

[0018] In various embodiments, the method may comprise weighting the map geographical features and / or the image geographical features. By weighting means assigning a factor or probability that represents its weight during a matching process. A higher weight meaning higher impact, and lower weight meaning lower impact. The probability may describe the probability that a feature has been correctly determ ined / identified.

[0019] This is advantageous because it enables the matching process to be adaptive to varying circumstances that would be cause for uncertainty in the matching. For example, some features are less permanent than others. For example, buildings and tree lines could be altered or removed by e.g., fire. The weighting can then be used to give such features less impact on the matching process. Another example of the weighting is if there’s an inherent uncertainty in the determination of image geographical features or map geographical features due to the method (some features are harder to determine).

[0020] In various embodiments, the method may comprise performing image segmentation of the digital image into a segmented digital image.

[0021] In various embodiments, the method may comprise dividing the sectors into a plurality of subsectors. Each subsector having a heading interval that is smaller than, or equal to, and contained within, the sector heading interval of respective sector. The image value representative for each sector may thus represent a condensed representation of the image values representing each subsector. Such a condensed representation of a subsector may e.g., take the form of the width and / or height of a land or sea area, the difference in height, the ruggedness of the terrain, or the form of terrain, such as rock, forest, or sand. Such a division into subsectors is advantageous in that it reduces the sensitivity to errors in the image values that may arise when determining the image value at the subsector level.

[0022] In various embodiments, the method may comprise performing image rectification processing of the digital image to take into account known intrinsic parameters of a camera. Image rectification refers to correction of for example image rotation, defocus, spherical aberration, coma, astigmatism, field curvature, image distortion, cylindrical or spherical reprojection, or morphing the image.

[0023] Such a procedure provides an advantage of avoiding errors caused by variations in the digital image that are caused by external factors, such as the type of camera used and instability in camera.

[0024] In various embodiments, the method may comprise updating the set of map data to an updated set of map data. The updated set of map data comprising geographical coordinates of an updated area. The updated area being associated with the obtained estimate of the geographical coordinate for the first geographical position, and repeating the steps summarized above using the updated set of map data. The updated area may furthermore be smaller than the area. The geographical coordinates of the updated area may also have a finer geographical distribution than the geographical distribution of the geographical coordinates of the area.

[0025] This has the advantage that the matching process can be done using a smaller set of map data, and therefore faster, as compared to using the full set of map data. It also has the advantage that the geographical position can be determined with higher accuracy when using a finer geographical distribution, as compared to using a less fine geographical distribution.

[0026] In various embodiments, the method may comprise determining the updated area based on the obtained estimate of a geographical coordinate for the first geographical position and based on information that indicates direction and speed at which the digital image is recorded.

[0027] In this way, it becomes possible to exclude a large part of the geographical positions in the map data, as they are impossible to reach with the given direction and speed. Thus, the set of updated map data can be greatly decreased in size, as compared to the full set of map data. This has the advantage that the matching process can be done significantly faster, as compared to using the full set of map data. This is the case both when using a procedure of pre-processing to create a database of map geographical features, as well as when performing the matching process directly from the map data.

[0028] In a further aspect there is provided a system for determining a geographical position comprising processing circuitry configured to determine, for at least one geographical coordinate from a set of map data and using geographical information described by the map data, map geographical features that are within the line of sight of said geographical coordinate. The system is further configured to represent the determined map geographical features with a respective map value, obtain a digital image recorded at a first geographical position of a landscape visible at the first geographical position, identify image geographical features in the digital image, create a query data structure, said query data structure comprising image values that represent said identified image geographical features in the digital image. The system is further configured to perform a matching of the query data structure with the map values representing the map geographical features, thereby obtaining an estimate of a geographical coordinate for the first geographical position at which the digital image was recorded.

[0029] In a further aspect there is provided a non-transitory computer-readable storage medium having stored thereon instructions to cause the system as summarized above to execute the steps according to the method as summarized above. These further aspects provide effects and advantages that correspond to those summarized above in connection with the method according to the first aspect.

[0030] BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Further objects, features and advantages of the present invention will appear from the following detailed description of the invention, wherein embodiments of the invention will be described in more detail with reference to the accompanying drawings, in which:

[0032] Figure 1a schematically illustrates a system,

[0033] Figure 1 b schematically illustrates an image of a landscape,

[0034] Figure 2 is a flowchart of a method,

[0035] Figure 3a-b schematically illustrate a respective map,

[0036] Figures 4a-c schematically illustrate ray tracing, Figure 5a schematically illustrates an image of a landscape,

[0037] Figure 5b-c schematically illustrate image of the image in figure 5a, and Figure 6 schematically illustrates sectoring of the image in figure 5a.

[0038] DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0039] Figure 1a illustrates a system 100 for determining a geographical position that comprises processing circuitry 110 configured to determine, for at least one geographical coordinate 303 from a set of map data 300 and using geographical information described by the map data 300, map geographical features 304 that are within the line of sight of said geographical coordinate 303. The system 100 is further configured to represent the determined map geographical features 304 with a respective map value 142.

[0040] Referring also to figure 1 b, the system 100 is further configured to obtain a digital image 150 recorded at a first geographical position 302 of a landscape 160 visible at the first geographical position 302 and identify image geographical features 170 in the digital image 150. The system 100 is further configured to create a query data structure 131 , the query data structure 131 comprising image values 132 that represent the identified image geographical features 170 in the digital image 150 and perform a matching of the query data structure 131 with the map values 142 representing the map geographical features 304, thereby obtaining an estimate of a geographical coordinate 303 for the first geographical position 302 at which the digital image 150 was recorded.

[0041] As exemplified in figure 1a, the system 100 may thus comprise appropriately configured processing circuitry 110 in the form of a processor 111 , memory 112 and an input / output unit 113. A camera 120 may form part of the system 100 or be an external device and be configured to record the digital image 150 and provide the digital image 150 via the input / output unit 113 to the processing circuitry 110.

[0042] Figure 1 a further illustrates a non-transitory computer-readable storage medium 101 having stored thereon instructions to cause the system 100 to execute steps of a method illustrated in figure 2.

[0043] Turning to figure 2, figure 3a and with continued reference to figure 1 b, a method of determining a geographical position will be exemplified. The method comprises determining 202, for at least one geographical coordinate 303 from a set of map data 300 and using geographical information described by the map data 300, map geographical features 304 that are within the line of sight of the geographical coordinate 303, for example, features visible in a landscape or seascape such as water, land, land edge, distance to land edge, height of land, fixed objects such as buoys and buildings.

[0044] The geographical position 302 may be a three-dimensional position comprising three coordinates such as latitude, longitude, and height, or any similar system. Alternatively, the geographical position 302 may be a two-dimensional position comprising 2 coordinates such as latitude and longitude, or any similar system.

[0045] In a representing step 203, the determined map geographical features 304 are represented with a respective map value 142. It is to be understood that the term “representing” may entail temporary storage of the map value 142. For example, the step of representing 203 the map geographical features 304 may comprise arranging the map values 142 in a vector-like map data structure 141.

[0046] A digital image 150 is obtained in an obtaining step 204, the digital image 150 having been recorded at a first geographical position 302 of the landscape 160 visible at the first geographical position 302.

[0047] The digital image 150 may be collected by any type of camera. Different types of cameras may be used for different circumstances. For example, the camera may be a short wave-infrared (SWIR), near infrared (NIR), or Thermal infrared (TIR). A combination of several types of cameras may be used to collect the digital image 150.

[0048] Thus, the method of determining the geographical position 302 may improve its performance by more reliably identifying image geographical features 170 under varying imaging conditions such as daytime, nighttime, and weather conditions such as fog, rain, and clouds.

[0049] In an identifying step 205 image geographical features 170 are identified in the digital image 150.

[0050] In a creating step 206, a query data structure 131 is created, the query data structure 131 comprising image values 132 that represent the identified image geographical features 170 in the digital image 150.

[0051] A matching procedure is performed in a matching procedure step 207 wherein a matching of the query data structure 131 is performed with the map values 142 that represent the map geographical features 304. An estimate of a geographical coordinate 303 is thereby obtained for the first geographical position 302 at which the digital image 150 was recorded.

[0052] In some embodiments, the query data structure 131 may be created from a plurality of digital images 150 such that the query data structure 131 may comprise image values 132 representing image geographical features 170 identified in the plurality of digital images 150. The plurality of digital images 150 may be taken by one camera or by a plurality of cameras. The plurality of cameras may be different types of cameras as previously described. The plurality of images 150 may be taken at different geographical positions. The plurality of digital images 150 may partially overlap or fully overlap.

[0053] In other words, the plurality of digital images 150 may depict different aspects and image geographical features 170 of the same landscape 160. By creating the data structure 131 from a plurality of digital images 150, more geographical image features 170 may be identified and / or a probability that the identified image geographical features 170 are correctly identified may be increased. For example, obstructions in the image such as water drops or dirt on the camera lens, animals, humans, or mobile objects which may be present only in some of the plurality of digital images 150, and therefore be compensated for when determining image geographical features 170. Some cameras may be able to detect image geographical features 170 that can not be detected by other types of cameras, for example when the image geographical features 170 are hidden behind clouds, smoke, rain or the like.

[0054] In some embodiments, the plurality of digital images 150 may be obtained 204 at different points of time and at different geographical positions. This is particularly useful for detecting, and excluding from the step of identifying 205 image geographical features 170 in the digital image 150, mobile objects that are not likely to have a corresponding map geographical feature 304. The mobile objects may be for example animals, humans, rain drops or dirt moving on the lens of the camera, vehicles or any object that is not expected to be part of the map geographical features of the set of map data 300.

[0055] In some embodiments, if a portion of the digital image 150 is obstructed, a portion of the query data structure 131 comprising the image values 132 that correspond to said obstruction may be omitted. Alternatively, one or several other digital images, that may originate from other cameras than the one used to obtain the digital image 150, may be used to reconstruct said obstructed portion and reveal the image geographical features 170 hidden behind the obstruction. An obstruction in a portion of the digital image 150 may be identified for example by comparing the plurality of digital images 150 obtained by the same camera at different points of time and / or obtained by a plurality of cameras that may be different types of cameras.

[0056] As illustrated in figure 2, in some embodiments, the determination step 202 may entail a creating step 2021 where a database 140 is created, the database 140 containing a plurality of geographical coordinates 303 associated with an area 301 described by the map data 300. Map values 142 that have been determined for respective geographical coordinate 303 in the determination step 202 are then stored in the database 140. In some embodiments, the database 140 may be a vector-like database. With this term, we here mean a structured database that allows for efficient searching with respect to several fields. The matching procedure may comprise providing the query data structure 131 to the database 140 to obtain an estimate of a matching map geographical features 304 in the database 140.

[0057] Referring also to figures 4a, 4b and 4c, the step of determining 202 the map geographical features 304 that are within the line of sight of said geographical coordinate 303 may comprise performing ray tracing 221 along headings 422 with respect to said geographical coordinate 303 and thereby detecting the presence of map geographical features 304 along each ray traced heading 422. By heading is meant an angle in one or more planes. For example, the map values 142 may be arranged such that the position of each map value 142 within the vector-like map data structure 141 is representative of a respective heading 422 with respect to said geographical coordinate 303 at which the map geographical feature 304 falls within the line of sight 410. With the term vector-like map data structure, we here refer to an ordered representation containing map-based data. In some embodiments, the step of representing 203 the determined map geographical features 304 may comprise weighting the map values 142 with a probability that a determined map geographical features 304 is correctly determined.

[0058] Map data is thus processed, for example by using ray tracing, based on a set of grid-points selected from possible positions. That is, a grid point corresponds to a geographical coordinate 303 of the map data 300 as described herein. For example, possible positions may be positions at sea or in the air where a vessel can possibly be (for a boat, given sufficient water depth, for a drone, above land, etc). Each grid point, i.e. , geographical coordinate 303, forms a candidate position along with a set of candidate headings. The notion of a grid position along with a heading may be referred to as a pose. For each pose, a map signature is created. That is, map geographical features 304 are represented by values 142 and the map signatures consist of such values 142. The map signature comprises values that represent map geographical features within the field of view of the respective grid point. Each signature aims at forming the same representation as may be determined from the image.

[0059] In order to determine the map geographical features 304 visible from each grid point 303, a ray is traced for a set of headings, where each heading can have either one or two components. Then, when a ray hits an object, which each is attributed a geographical feature, that geographical feature is saved for that heading. Each ray thus results in a level 1 feature. Then, from all level 1 features, a set of level 2 feature vectors, one vector per possible heading and azimuth, are created. The set of level 2 feature vectors is thus a condensed representation of what is expected to be seen from one position, in a specific direction. In figure 4a, this is exemplified for a single azimuth. In figure 4b, the level 2 feature “land height” is illustrated, which is computed after the ray tracing is complete. In figure 4c, it is illustrated that a variety of level 2 features can be created, given the level 1 features received from the ray tracing step. The notion of level 1 and 2 features will further be clarified in the following sections, with more examples.

[0060] Each ray may comprise a plurality of sampling points distributed with a spacing interval along the heading 422. Tracing the ray thus implies determining whether a map geographical feature 304 is present at least one of the plurality of sampling points. Determining whether a map geographical feature 304 is present at at least one of the plurality of sampling points may be done for one sampling point at a time, multiple sampling points at the same time, or all sampling points of the corresponding ray at the same time, in any order or consecutively in a direction along the heading 422. That is, determining the presence of a map geographical feature 304 along the ray implies the map geographical feature 304 is determined to be at one of the plurality of sampling points of the corresponding ray. In some embodiments, the spacing interval of the plurality of sampling points of each ray may vary. In this way, computational power and time of the method may be reduced by increasing the spacing interval in cases where it is not considered necessary, for example if no variation in the map geographical features 304 of the map data 300 are expected along the corresponding heading 422. When more detailed tracing is considered necessary, the spacing interval may correspondingly be decreased in order to enable determination of more variation in the map geographical features 304 of the map data 300 along the corresponding heading 422.

[0061] In some embodiments, each ray may have a width and a height. In this case, the plurality of sampling points of each ray comprises a sampling area. The sampling area corresponds to a cross-section of the ray, orthogonal to the heading 422 of the ray, at the location of the corresponding sampling point along the heading 422 of the ray. Each ray may have a different width and height.

[0062] In other words, the sampling area of each ray may be adapted as considered necessary. By tuning the sampling area, the efficiency of the method in terms of computational time and power may be tuned. Adapting the sampling area may also enable more accurate determination of a distance from the first geographical position 302 to the corresponding determined map geographical feature 304.

[0063] In some embodiments, ray tracing from the first geographical position 302 may not be performed for regions of the map data 300 where ray tracing from a previous geographical position have been performed and where it was determined that no map geographical feature 304 is present in said region of the map data 300.

[0064] Land at the edge of the water, or land edge, is an example of a level 2 feature and are retrieved by first calculating the difference in distance before first land hit between two adjacent rays (That is, the relative distance between the closest land mass for the two rays.). If the distance is large enough, i.e. , exceeding a selected threshold value, it is determined that there lies a land edge in that heading of the field of view. Further, it has been found that some land edges are difficult to detect while other edges are easy to determine. With the map data it is possible to predict what land edges are easy to see and which ones are harder. Most land edges are either detectable or not, but some are in between, and this is a way to handle that.

[0065] Hence, in order to determine a probability of detection of such an edge, detectability, the vertical field of view difference of the land edge with respect to the background is used, as well as the background, i.e. , if the background is open water the edge is generally easier to detect. Conversely, if the edge has a distinct field of view difference with its background, it is easily detectable from an image.

[0066] Figure 4a shows an example position of grid point and feature extraction from the map, i.e., Figure 4a illustrates determination of map geographical features 304 by ray tracing. The visible land points, as well as visible land edges, for each heading in the field of view have been outlined according to the ray tracing. X’s mark the left land edges, and O’s mark right land edges.

[0067] In order to determine what relevant objects apart from land edges are visible from the candidate position, i.e., from each geographical coordinate 303, the ray may also be traced in the reverse direction. The ray starts at the relevant object and goes towards the candidate position, i.e., towards the relevant geographical coordinate 303. If the ray hits land, it is determined using height data whether it is occluded or not. In order to determine occlusion, rays are traced using height data available from the map data 300. In Figure 4c, the angle 02 is larger than the angle ai , and the only information saved for this ray is thus the angle 02. The detectability of such objects is determined by distance and predictable occlusions. If the detectability is zero due to occlusion, the object is omitted.

[0068] Further clarifying the concept of “Level 1” feature, a vector or array holding level 1 features may be a representation of heading classification. For a pose, a set of headings is visible, given the camera 120 and its field of view. Each of the headings is classified in a set of aspects. Does the heading contain land? Does the heading contain a house? If it contains land, how high is that land? Each of the aspects has its own level 1 feature vector representation, which can be easily retrieved from a camera in real life, but also with preprocessing of the map as is the case in some embodiments.

[0069] Similarly, the concept of “Level 2” feature is to form a condensed representation of a subsector. A level 2 feature represents a characteristic of the subsection, such as, e.g., the variation in height of a land mass, the maximum or minimum slope of land, the width and / or height of a land or sea area, the texture of the area, such as rock, forest, or sand. The level 2 features may be considered as existing in a unified domain and as such are easily matchable between image and map. For each feature, one may also include a measure of the expected accuracy of said feature, to allow for different levels of significance between found features.

[0070] The image geographical features 170 are identified in the identification step 205 by the processing of the image 150 recorded by the camera 140 with known intrinsic parameters as well as any preprocessing such as stabilization. Preferably the placement of the camera 120 is known (although, this is not required as one may determine this using, e.g., earlier images, or may even just be set as a given fixed value). Compass information may be used such that the heading of the camera is known, although this is not required, as one may determine this from, e.g., earlier images or in relation to the heading.

[0071] The step of identifying 205 image geographical features 170 in the digital image 150 and the step of creating 206 a query data structure 131 comprising image values 132 that represent the identified image geographical features 170 may comprise arranging the image values 132 in a vector-like data structure. Similar to the map values 142 and map signatures discussed above, the image values 132 may be defined in terms of an image signature. That is, image geographical features 170 are represented by image values 132 and the image signature consists of such image values 132.

[0072] The step of identifying 205 image geographical features 170 may comprise performing image segmentation of the digital image 150 into a segmented digital image 500. That is, to compute the image signature, the image 150 may be processed by two or more deep neural networks. Figure 5a illustrates the digital image 150 before segmentation and Figures 5b and 5c illustrate the output of the two or more neural networks after segmentation. As illustrated in Figure 5b, the first neural network returns the semantic segmentation of land mass 171 and, as Figure 5c illustrates, the second neural network returns bounding boxes 172 for objects such as buildings and lighthouses that are relevant for determining the position. Further networks may also be included in this process.

[0073] Because the images are typically recorded with a camera that is not perfectly modeled by a pinhole camera, it may be relevant to perform image rectification processing of the digital image 150 to take into account known orientation and intrinsic properties of the camera 120. Such image rectification processing may comprise correction of any of image rotation, defocus, spherical aberration, coma, astigmatism, field curvature, image distortion, etc. The semantic segmentation is thus rectified with the help of information from the known intrinsic of the camera, e.g., focal length, principle point, and a model for the radial distortion. Typically, correction in the form of image rotation will be performed in order to correct for a non-horizontal horizon line in the images since the camera 120 may in many instances be orientated such that the horizon line will not be accurately aligned with the orientation of a two-dimensional image sensor within the camera 120.

[0074] Moreover, the creation in creating step 206 of the query data structure 131 may comprise weighting the image values 132 with the probability that the image geographical feature 170 determined during the image segmentation is correctly identified.

[0075] As mentioned above, the step of creating 206 the query data structure 131 comprising image values 132 that represent the identified image geographical features 170 may comprise arranging the image values 132 in a vector-like data structure. This means that the segmentation into land mass may be converted to a height vector, h:

[0076] The height vector, h, is a representation of the land silhouette, where each position in the vector corresponds to a heading in the field of view of the camera 120. Each position in the vector, holds the field of view, detected in that heading, which is determined with the help of the segmentation. In other words, each position of the vector corresponds to a heading in the field of view, and each image value 172 at each position of the vector corresponds to a perceived height of the land mass in that heading. A distance vector may be constructed in an analogous way. For example, distance values may be determined by approximating in the same way as a human can approximate the distance, i.e. , a neural network can be trained to approximate the distance.

[0077] The output of the second, object detecting, neural network is bounding boxes for all objects 172 relevant for the determination of the geographical position. For example, these objects 172 may be defined by different classes such as edges of land mass, positioning marks, buoys, masts or houses. The bounding boxes are rectified using the intrinsic camera parameters. Each class is represented by a vector, c, for class i, where each position of the vector represents a heading:

[0078] Each position thus corresponds to a heading in the horizontal field of view, and each vector position holds a value indicating the level of trust or strength of the found feature. For example, a value of one if an object of the class is detected in that heading and a value of zero otherwise. If an object is detected in the heading that corresponds to some vector position, the element c^j is equal to one. Other values, for example between 0 and 1 , may be used.

[0079] As illustrated in Figures 5b, 5c and 6, the creation in step 206 of the query data structure 131 may comprises dividing 2061 the segmented digital image 500 into a plurality of sectors 610, where each sector 610 represents a sector heading interval 602 of the digital image 150. The respective image values 132 identified in each sector 610 may be arranged in an arranging step 2062 such that the position of each image value 132 within the query data structure 131 is representative of the respective sector heading interval 602. Furthermore, as illustrated in figure 6, the sectors 610 may be divided into a plurality of subsectors 612, each subsector 612 having a subsector heading interval that is smaller than, and contained within, the sector heading interval 602 of the sector 610. The image value 132 that represent the image geographical features 170 identified in each sector 610 may then be a condensed representation of the values representing each subsector 612.

[0080] In other words, such a segmentation of the heading by a set of predetermined sectors may be represented by a feature function, f2:

[0081] In f2, each feature (i.e. , values of the h and c vectors as described above) is summed by a predetermined set of sectors 610, defined by k and I, where k is the sector width and / is the stride of which the sector 610 is displaced for each position in the output vector f2. That is, every sector 610 is assigned a number for each feature. This sectoring procedure reduces the sensitivity to small errors in the determination of the values 142, 132 in the h and c vectors. The sectors 610 can be constructed in many ways, and preferably overlapping. In the example in Figure 6, six non-overlapping sectors 610 are displayed, though a more complex sector pattern is possible. This is just a simple example embodiment of a level 2 feature function to further give context to the concept.

[0082] While processing the map data 300 and creating vectors for each possible geographical coordinate 303, the vectors may be stored in a special data structure: a vector database. A vector database is a data structure specialized in searching vast amounts of data with a query vector. Given a query vector, it returns the closest vectors that are already in it. Further, belonging to each inserted query vector, a position is stored. Hence, in the present context, the procedure of matching the found image features 170 to the features 304 of different potential geographical positions allows for a ranking of these positions, allowing both for probability-based tracking system for a sequence of images and for an interpolation in between candidate map positions to allow for higher positioning accuracy than that given by the used map grid. Once a match has been found, defined by the highest value of similarity exceeding a predetermined threshold, and given that the match is reasonable according to the parameters of the previous position, the current position is returned.

[0083] In some embodiments, as illustrated in Figure 3b, the set of map data 300 may be updated to an updated set of map data 370, the updated set of map data

[0084] 370 comprising geographical coordinates 373 of an updated area 371 associated with the obtained estimate of the geographical coordinate 303 for the first geographical position 302, and repeating the steps 202, 203, 204, 205, 206, and 207 using the updated set of map data 370. The updated area 371 may be smaller than the area 301 and the geographical coordinates 373 of the updated area 371 may have a finer geographical distribution, i.e., grid, than a geographical distribution of the geographical coordinates 303 of the area 301 . For example, the updated area

[0085] 371 may be determined based on the obtained estimate of a geographical coordinate 303 for the first geographical position 302 and based on information that indicates direction and speed at which the digital image 150 is recorded. In other words, given that the grid cannot be of infinite resolution, active interpolation ray tracing may be used in order to improve the resolution. When having a grid with grid points with distance d in between the grid points, and retrieving a geographical coordinate that is a strong candidate position, the same map preprocessing that has been done offline may be performed for a set of new potential geographical coordinates with higher resolution to allow for a more accurate matching, thereby yielding a higher resolution positioning. The thus formed candidate positions are then matched with the image features together with the already existing position candidates. This process may then be repeated to determine highly accurate positioning estimates.

[0086] As mentioned above, in some embodiments, the step of representing 203 the determined map geographical features 304 may comprise weighting the map values 142 with the probability that a determined map geographical feature 304 is correctly determined. Because some map geographical features 304 are more important than others, there is a need to weight them before storing the corresponding values 142 in the database 140. This is done by multiplying each map geographical features 304 with a weight before insertion. In this way, the distance in the feature space is more dependent on the heavily weighted features than others. Alternatively, a second feature indicating the reliability of the map geographical features 304 may be stored to indicate the expected reliability of the map geographical features 304 . As an illustrative example, the contour information of a pose (i.e. , a geographical coordinate and associated heading) may be deemed less reliable if the height information is predominantly due to buildings, structures, or foliage that may easily be affected by, e.g., fire, such that this information may be deemed to potentially differ from expectation, making it less reliable than height information due to land formations that are difficult to change. The former information may then be deemed less reliable than the latter, and being stored in the database correspondingly.

[0087] In some embodiments, an image feature weight that describes the probability that the identified image geographical feature 170 is correctly identified may be dependent on the image geographical features 170 that have been identified in one or more of a plurality of previous digital images 150 taken at one or more previous geographical positions. The image feature weight may for example depend on the type of image geographical features 170 identified in one or more of the plurality of previous digital images 150; the number of times a type of image geographical feature 170 is identified in one or more of the plurality of previous digital images 150; the distance from the corresponding previous geographical position to the map geographical feature 304 of the corresponding identified image geographical feature 170 in one or more of the plurality of previous digital images 150; the corresponding image feature weight of each identified image geographical feature 170 in one or more of the plurality of previous digital images 150, and to what extent the identified image geographical features 170 agrees with the identified image geographical features 170 in one or more of the plurality of previous digital images 150.

[0088] In other words, the probability that the identified image geographical feature 170 is identified correctly, i.e. the image feature weight, may depend on for example how close the map geographical feature corresponding to the image geographical feature 170 is to the first geographical position 302, if the first geographical position 302 is close to or far from land, and / or if similar object were identified at previous geographical positions in previous digital images.

[0089] In some embodiments, the estimated geographical coordinate 303 for the first geographical position 302 may comprise a positioning confidence. The positioning confidence may correlate with the image feature weight for each corresponding image geographical feature 170 identified in the digital image 150 recorded at the first geographical position 302.

[0090] Depending on if the positioning confidence is below or above a confidence threshold, the method may comprise performing one or more of the following: updating the set of map data 300 to the updated set of map data 370; changing the distribution of geographical coordinates 303 (e.g. a density of geographical coordinates 303); changing the number of headings 422 that are ray traced from the first geographical position 302 and an angular spacing between the headings 422; changing the number of sectors 610 of the segmented digital image 500 and / or the sector heading interval 602; changing the number of digital images 150 from which the query data structure 131 is created; and in the case that the query data 131 is created from a plurality of digital images 150 obtained at different points of time, an image frequency describing the time lapsed between obtaining each of the plurality of digital images 150 may be changed.

[0091] In other words, depending on the positioning confidence, the method can be adapted to comprise more or fewer computations to arrive at a new positioning confidence. By for example, using a larger updated area 371 of the updated map data 370, reducing the sector heading interval 602, and / or increasing the density of geographical coordinates 303, the positioning confidence may be increased. Alternatively, if a reduction in positioning confidence is desired (in order to reduce computational time and power), said features may instead be decreased such that the computational power and time required may be decrease. In some cases, if the positioning confidence is relatively high, the density of geographical coordinates may be increased in a vicinity of the first geographical position 302 to enable a more precise determination of the first geographical position 302.

[0092] In some embodiments, the sector heading interval 602 and / or the number of sectors 610 may be increased in order to enable identifying 205 smaller image geographical features 170 in the segmented digital image 500. A smaller image geographical feature 170 may imply the image geographical feature 170 is further away from the first geographical position 302 and / or that the image geographical feature 170 is relatively smaller. The sector heading interval 602 may vary in different regions of the segmented digital image 500, for example if one region is expected to have more variation in image map geographical features 170 (e.g. land, compared to open water) or smaller image geographical features 170. Similarly, the number of headings 422 that are ray traced from the first geographical position 302 may be increased in order to enable determining map geographical features 304 further away from the first geographical position 302. The angular spacing between the headings 422 may vary in different angular regions, for example if one angular region is expected to have more variation in map geographical features 304, e.g. land, than another region, e.g. open water.

[0093] In some embodiments, the step of performing 207 a matching of the query data structure 131 with the map values 142 can be modified depending on the positioning confidence. For a relatively low positioning confidence, the calculation complexity of the step of performing 207 a matching may be increased in order to increase the positioning confidence by performing any of the previously described adaptions that can be attributed to increasing the positioning confidence. For a relatively high positioning confidence, the calculation complexity of the step of performing 207 a matching may be decreased in order reduce computational time and power by performing any of the previously described adaptions that can be attributed to reducing computational time and power.

[0094] When matching the signatures, i.e., the matching procedure involving the map values 142 and the image values 132, one obtains a list of geographical coordinates that are likely candidate positions. These are retained as a set; the most likely candidate is then selected by combining the fitting with the earlier candidate lists using a Viterbi-style algorithm. In such a Viterbi-style algorithm, at each time, a set of candidate positions are determined using the image observed at that time; between each such state, the likelihood of a transition between different states are formed using the travel distance between candidate positions. Any position that is unreachable given the current velocity may then be omitted, making this transition impossible. The determined likelihoods for each state may also be updated using current information, and in this sense correct earlier assumed likelihoods. The state trellis is in this way updated for each image, retaining a collection of the states from earlier images; in this way, the positioning is based on the current image and the earlier states, allowing also for a measure of the certainty of this positioning estimate.

[0095] The system 100 for determining a geographical position may be configured to perform the method according to any of the described embodiments.

[0096] In some embodiments, the system 100 may be integrated in ensemble comprising one or more cameras that combined may give a 360 degree view. The ensemble may comprise a sensor such as a magnetometer, accelerometer, and gyroscope. The ensemble may be incorporated in or attached to a vehicle such as an unmanned surface vehicle (USV) or an unmanned aerial vehicle (UAV).

[0097] In some embodiments, the system 100 may be integrated in a phone wherein the phone comprise one or more cameras that may obtain 204 the digital image 150.

[0098] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" "comprising," "includes" and / or "including" when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0099] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0100] The foregoing has described the principles, preferred embodiments and modes of operation of the present invention. However, the invention should be regarded as illustrative rather than restrictive, and not as being limited to the particular embodiments discussed above. The different features of the various embodiments of the invention can be combined in other combinations than those explicitly described. It should therefore be appreciated that variations may be made in those embodiments by those skilled in the art without departing from the scope of the present invention as defined by the following claims.

Claims

CLAIMS1 . A method of determining a geographical position, the method comprising:- determining (202), for at least one geographical coordinate (303) from a set of map data (300) and using geographical information described by the map data (300), map geographical features (304) that are within the line of sight of said geographical coordinate (303);- representing (203) the determined map geographical features (304) with a respective map value (142);- obtaining (204) a digital image (150) recorded at a first geographical position (302) of a landscape (160) visible at the first geographical position (302);- identifying (205) image geographical features (170) in the digital image (150);- creating (206) a query data structure (131 ), said query data structure (131 ) comprising image values (132) that represent said identified image geographical features (170) in the digital image (150); and- performing (207) a matching of the query data structure (131 ) with the map values (142) representing the map geographical features (304), thereby obtaining an estimate of a geographical coordinate (303) for the first geographical position (302) at which the digital image (150) was recorded.

2. The method according to claim 1 , comprising:- creating (2031 ) a database (140) containing a plurality of geographical coordinates (303) associated with an area (301 ) described by the map data (300) and storing map values (142) for respective geographical coordinate (303) in the database (140).

3. The method according to any of the previous claims, wherein the step of representing (203) the map geographical features (304) comprises arranging the map values (142) in a vector-like map data structure (141 ).

4. The method according to claim 3, wherein the step of determining (202) the map geographical features (304) that are within the line of sight of saidgeographical coordinate (303) comprises performing (2021 ) ray tracing along headings (422) with respect to said geographical coordinate (303) and detecting the presence of map geographical features (304) along each ray traced heading (422).

5. The method according to claim 4, comprising arranging the map values (142) such that the position of each map value (142) within the vector-like map data structure (141 ) is representative of a respective heading (422) with respect to said geographical coordinate (303) at which the map geographical feature (304) falls within the line of sight (410).

6. The method according to claim 5, wherein the representing (203) the determined map geographical features (304) comprises weighting the map values (142) with a probability that a determined map geographical feature (304) is correctly determined.

7. The method according to any of the previous claims, wherein the step of creating (206) a query data structure (131 ) comprises arranging the image values (132) that represent the image geographical features (170) in a vectorlike data structure.

8. The method according to any of the previous claims, wherein the identifying (205) of image geographical features (170) comprises performing image segmentation (2051 ) of said digital image (150) into a segmented digital image (500).

9. The method according to claim 8, wherein creating (206) the query data structure (131 ) comprises:- dividing (2061 ) the segmented digital image (500) into a plurality of sectors (610), where each sector (610) represents a sector heading interval (602) of the digital image (150); and- arranging (2062) the respective image values (132) identified in each sector (610) such that the position of each image value (132) within the query data structure (131 ) is representative of the respective sector heading interval (602).

10. The method according to claim 9, wherein creating (206) the query data structure (131 ) comprises:- dividing the sectors (610) into a plurality of subsectors (612), each subsector (612) having a subsector heading interval that is smaller than, and contained within, or equal to, the sector heading interval (602) of said sector (610); and wherein the image value (132) that represent the image geographical features (170) identified in each sector (610) is a condensed representation of the values representing each subsector (612).

11. The method according to any of the previous claims, comprising performing image rectification processing of the digital image (150) to take into account known intrinsic properties of a camera (120), the image rectification processing comprising correction of any of:- image rotation,- defocus,- spherical aberration,- coma,- astigmatism,- field curvature, and- image distortion.

12. The method according to any of the claims 8 to 11 , wherein creating (206) a query data structure comprises weighting the image values (132) with a probability that the image geographical feature (170) determined during the image segmentation is correctly identified.

13. The method according to any of the previous claims, where performing (207) a matching comprises providing the query data structure (131 ) to the database (140) to obtain an estimate of a matching map geographical features (304) in the database (140).

14. The method according to any of the previous claims, comprising updating (208) the set of map data (300) to an updated set of map data (370), the updated set of map data (370) comprising geographical coordinates (373) ofan updated area (371 ) associated with the obtained estimate of the geographical coordinate (303) for the first geographical position (302), and repeating the steps (202, 203, 204, 205, 206, 207) according to claim 1 using the updated set of map data (370).

15. The method according to claim 14, wherein the updated area (371 ) is smaller than the area (301 ) and the geographical coordinates (373) of the updated area (371 ) have a finer geographical distribution than a geographical distribution of the geographical coordinates (303) of the area (301 ).

16. The method according to claim 14 or claim 15, wherein the updated area (371 ) is determined based on the obtained estimate of a geographical coordinate (303) for the first geographical position (302) and based on information that indicates direction and speed at which the digital image (150) is recorded.

17. The method according to any of the previous claims, where the determining (202) of map geographical features (304) comprises identifying features such as water, land, land edge, distance to land edge, height of land, fixed objects such as buoys and buildings.

18. A system (100) for determining a geographical position comprising processing circuitry (110) configured to:- determine, for at least one geographical coordinate (303) from a set of map data (300) and using geographical information described by the map data (300), map geographical features (304) that are within the line of sight of said geographical coordinate (303);- represent the determined map geographical features (304) with a respective map value (142);- obtain a digital image (150) recorded at a first geographical position (302) of a landscape (160) visible at the first geographical position (302);- identify image geographical features (170) in the digital image (150);- create a query data structure (131 ), said query data structure (131 ) comprising image values (132) that represent said identified image geographical features (170) in the digital image (150); and- perform a matching of the query data structure (131 ) with the map values (142) representing the map geographical features (304), thereby obtaining an estimate of a geographical coordinate (303) for the first geographical position (302) at which the digital image (150) was recorded.

19. A non-transitory computer-readable storage medium (101 ) having stored thereon instructions to cause the system (100) according to claim 18 to execute the steps according to any one of claims 1 to 17.

Images