System and method of georeferencing two images with one another

Abstract

A method of georeferencing a raster image using a comparison of an already- georeferenced raster image. The method may further include performing a gridline transformation on a portion of a cell of the raster image to position, scale and align a portion of a raster image, which is suitable for facilitating georeferencing hand-drawn images.

Description

[0001] SYSTEM AND METHOD OF GEOREFERENCING TWO IMAGES WITH ONE ANOTHER

[0002] Field of the Invention

[0003] The present disclosure relates to improvements in platforms, systems and methods to accurately scale and align a digital map image with an image associated with geographical coordinates.

[0004] Background of the Invention

[0005] Images of floorplans are often difficult to georeference due to the following issues:

[0006] 1 . The image might be a badly taken photo of the floorplan with skewing / perspective distortion.

[0007] 2 . The floorplan might not be drawn to scale / have correct proportions.

[0008] 3 . Features in the floorplan might not correspond to features in the outdoor image.

[0009] For example, roof overhangs make it hard to identify where exterior walls are on an overhead / outdoor image. A similar situation exists for buildings with architectural features or cladding that cause their indoor footprint to be smaller than their outdoor footprint.

[0010] The first and second points impact simplistic georeferencing methods that only offer translation / rotation / scaling of the entire floorplan. If a floorplan is not drawn with the correct proportions, then there is little chance of rescaling the floorplan in a way that will ever get it to line up with the corresponding building footprint on a map.

[0011] The third point severely impacts georeferencing methods that use feature points.

[0012] These methods use common points in both images to calculate the required transforms. Without easily identifiable common points, automated methods cannot be applied, and the best possible method is for a human to make a best-effort guess about the locations of certain features. Because feature-point based geotagging does not separate the operations of translation / rotation / scaling, any errors in the placement of feature points result in errors to all three dimensions. Accordingly, there exists a need to provide an improved system less prone to error generation.

[0013] The present disclosure seeks to lessen these problems by providing a system which allows positioning, rotation and scaling independently without significant errors creeping in. It will be clearly understood that, if a prior art publication is referred to herein, this reference does not constitute an admission that the publication forms part of the common general knowledge in the art in Australia or in any other country.

[0014] Summary

[0015] In one or more preferred aspects, the present disclosure describes a method that considers the imperfect nature of real-world floorplan imagery, and handles up to all three issues identified above.

[0016] 1 . The “straighten image” operation allows for the straightening and correction of perspective for photos of floorplans.

[0017] 2. The “gridline transform” operation allows floorplans to be conformed and distorted to ensure that their proportions match the real shape of the building.

[0018] 3. The separation of positioning / rotation / scaling operations is extremely useful for “best-effort” situations where there are not common feature points.

[0019] It is typical for a floorplan to have some kind of scale indicator on them (this could be a measurement drawn onto the plan, or a scale bar, or even the use of a common width such as the size of a door), making it easy to correctly scale the floorplan. Although roof overhangs can make it hard to identify where the walls are, their edges are typically parallel to the walls of the building which makes it easy to correctly rotate the floorplan. The only operation that cannot typically be done correctly when there are not common points is the positioning. Fortunately, errors in positioning in one or more aspects of the methods described below do not impact scaling or rotation, so the floorplan will have the correct internal area / shape, which is typically more important than being perfectly positioned.

[0020] The present disclosure in one preferred aspect provides for a method of georeferencing two raster images with one another. The method includes providing a georeferenced outdoor image and an ungeoreferenced image of interest, the image of interest being at least in part transparent and transforming geographical coordinates associated with the outdoor image into two-dimensional coordinates suitable for presentation on a screen display. The method further includes independently and separately: positioning the image of interest over the outdoor image; scaling the image of interest to dimensionally coincide with the outdoor image; and aligning the image of interest with the outdoor image to create an aligned image. The method further includes inversely transforming the aligned image to geographical coordinates to georeference the aligned image.

[0021] The present disclosure in another preferred aspect provides for a method of performing a nested guideline transformation of a portion of a cell of a raster image without affecting another portion of the raster image. The method includes selecting a portion of the raster image; differentially positioning and scaling the portion of the raster image compared to a remainder of the raster image; and aligning the differentially positioned and scaled portion with the remainder of the raster image.

[0022] One or more techniques described herein are applicable to the creation of map data. When creating map data, it is typical to work with lower-quality sources than the final output. This can include satellite imagery, scans of original architectural drawings, scans of hand-drawn floorplans annotated with notes, overhead drone photos, etc. These sources are typically raster images; not vector data. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed. In the present specification and claims, the word “comprising” and its derivatives including “comprises” and “comprise” include each of the stated integers, but does not exclude the inclusion of one or more further integers.

[0023] It will be appreciated that reference herein to “preferred” or “preferably” is intended as exemplary only. The claims as filed and attached with this specification are hereby incorporated by reference into the text of the present description. The disclosure of International Patent Publication No. WO 2020 / 163913, titled “System and Method For Indoor Spatial Mapping” is incorporated by reference herein in its entirety.

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several embodiments of the invention and together with the description, serve to explain the principles of the invention.

[0025] Brief Description of the Figures

[0026] Fig. 1 is a diagrammatic flow of a method of gridline transformation in accordance with a preferred embodiment of the present disclosure.

[0027] Fig. 2 shows a hand-drawn source image useable with a gridline transformation to provide a more accurate scaling of building proportions.

[0028] Fig. 3 shows the hand-drawn proportions in the image of Fig. 2 compared to the real dimensions of the building, shown in dashed outline.

[0029] Fig. 4 shows the image of Fig. 3, improved by non-uniform scaling.

[0030] Fig. 5 shows the image of Fig. 2 after applying a gridline transformation. Fig. 6 shows a progressive dragging of horizontal and vertical gridlines over the source image of Fig. 2.

[0031] Fig. 7 shows a progressive dragging of the gridlines of Fig. 6 over a destination image.

[0032] Fig. 8 shows application of a nested transformation of a portion of the source image by intersecting gridlines.

[0033] Fig. 9 shows application of the nested transformation between the source image and the destination image.

[0034] Fig. 10 shows the results of a nested gridline transformation between the source image and the destination image.

[0035] Fig. 11 is a diagrammatic flow of a method of georeferencing in accordance with a preferred embodiment of the present disclosure.

[0036] Fig. 12 is an overhead perspective view of an outdoor image of a building.

[0037] Fig. 13 is a progressive view of the outdoor image of Fig. 12 with a reference line drawn along one side of the building, then the building reference line rotated to be horizontal position.

[0038] Fig. 14 shows a progressive flow of rotation and scaling of an internal building plan, with the building plan being progressively being scaled up (enlarged).

[0039] Fig. 15 is rotated view of the building plan of Fig. 14, being scaled up. Fig. 16 is a top plan progressive view of an example of rescaling using two reference points.

[0040] Fig. 17 shows progressive perspective views of a method of straightening a document in accordance with another preferred aspect of the present disclosure.

[0041] Detailed Description of the Drawings

[0042] Reference will now be made in detail to the present preferred embodiments of the disclosure, examples of which are illustrated in the accompanying drawings.

[0043] An enhanced and accurate method for georeferencing to images will now be described. As a preliminary step, two images are selected. A first, georeferenced image, typically an overhead outdoor or external image, is selected as a base image. The outdoor image may be a top plan or overhead perspective image of a building, terrain, city block area, and / or region. The outdoor image may be taken as an aerial photograph from a drone, plane, or satellite. Such images are typically associated with physical coordinates projected from a geographic coordinate system (GCS), such as a WGS84 or NAD83 spherical coordinate system (latitude and longitude). The outdoor image is then transformed to a projected coordinate system (PCS) to facilitate comparison with a digitally stored second image. The outdoor image is preferably transformed from a GCS to a PCS using a meters-based projection, such as a Web Mercator projection. Other map projections may be used, but may not correctly represent the shape of landmasses linearly. For example, circles get distorted into ovals. Web Mercator is a preferred projection as the distortion is linear (e.g., circles do not become ovals). Examples of other map projections include UTM, Robinson, Lambert, Sinusoidal, and Abers. The outdoor image is then transformed to a two-dimensional digital map image using a mathematical function such as an affine transformation to yield the image in terms of data as X and Y pixel coordinates, for viewing on an electronic display such as a computer monitor (laptop or hand-held device with a screen).

[0044] Once transformed for display on screen, the outdoor image may be rotated by a user to match the “natural rotation” of the building / city block / etc that is of interest. The “natural rotation” is one that causes most lines to become horizontal or vertical, rather than being shown at whatever rotation the building happens to be oriented and displayed. While this step is presented as an early step in the method to simplify the user’s experience when using a touchscreen, it is not a mandatory part of the process, and it may also be performed at a later point during the process.

[0045] A second, ungeoreferenced image is chosen, which here is termed as an “image of interest” to georeference features within the second image. The image of interest is typically obtained as a digital image already digitally stored and associated with X, Y pixel coordinates, for example, an already scaled building plan. However, the image of interest may be a hand-drawn image, which needs to be properly positioned, scaled, and aligned for eventual alignment with the first, georeferenced image to obtain an aligned georeferenced image.

[0046] Figs. 1 to 10 illustrate a preferred method of transforming the image of interest to ready the image of interest for alignment with an already georeferenced image. The digital transformations of the image of interest (positioning, scaling, rotation, straightening perspective, and distortion) are performed as discrete operations (for example, separate and independent operations). This ensures that errors in one dimension such as positioning do not cascade into errors with other dimensions such as scaling and rotation. This is important as with indoor mapping, accurate scaling is of much greater importance than accurate positioning. A slightly inaccurate positioning will not critically impact products created from the final georeferencing compared to inaccurate scaling, however inaccurate scaling can lead to inaccurate calculations of critical commercial metrics such as net lettable area.

[0047] Separately performing positioning and scaling (and other transformations) as their own operations results in a better georeferenced image. For example, in the case of the outdoor image being of a satellite view of a building and the image of interest being a floorplan, it may not be possible to accurately determine the exact positioning of the floorplan due to features in the outdoor image like roof overhangs, or due to the perspective of the outdoor image causing a vertical building to appear at an angle. However, it may be easy to determine the correct scale and rotation for the image of interest. In such cases, performing all transformations in a single step will lead to the propagation of positioning errors into the scaling of the image, even though correct positioning may not be as critical as correct scaling. Performing the transformations in as their own separate operations prevents this propagation of errors.

[0048] The separate transformation operations are grouped into four categories - distortion, rotation, scaling, and positioning. These transformations are typically performed in that order, however this is not a requirement, and some transformations may not be required at all.

[0049] First, the image of interest must be transformed to remove any distortions present in the image. One such transform is a “straightening” operation to remove any perspective / skew created by capturing the image of interest at an angle - the details of which would be appreciated by those of ordinary skill in the art. The other transform is a “gridline transformation” operation that allows for local distortions in the image to be removed by dragging corresponding horizontal and vertical gridlines over the outdoor image and image of interest. Figs. 1 -10 illustrate a gridline transformation in a portion of an image.

[0050] Referring to Fig. 2, local distortions are common of floorplans drawn “in the field,” where an accurately drawn outline of a building may then be annotated with hand drawings and measurements describing the contents of that building. Although the building outline is correctly drawn, the rooms and contents of the building are unlikely to be correctly sized if drawn by hand. Gridline transformation allows for these hand drawings to be brought into scale without affecting the already correctly scaled outline of the building.

[0051] Fig. 2 shows an example of an image 120 when the gridline transformation is applicable. Its hand-drawn nature and annotated measurements are typical of drawings made “in the field,” however this method can also be applied to other kinds of images. For example, digital floorplans directly created from a hand-drawn source may preserve the same inaccuracies present in the hand-drawn source and require the same type of transformation to correct them.

[0052] Fig. 3 shows the limits of uniform scaling when dealing with hand-drawn imagery - the proportions of a building may not be correct in the image compared to reality (illustrated by dashed lines 122). This can be improved by using non-uniform scaling (that is, scaling the image by different amounts horizontally and vertically) as shown in Fig. 4. While the outline of the building is correctly scaled on the exterior, the internal walls are not in alignment.

[0053] Fig. 5 illustrates the result of applying a gridline transformation, with all walls being brought into alignment with their real dimensions. Referring again to Figs. 1 and 6, the user first drags horizontal and vertical gridlines over the source image / image of interest. The same number of gridlines are then dragged over the destination image / outdoor image and aligned to the same reference points / features as the source image (see Fig. 7). Because the destination image is already correctly georeferenced, accurate measurements can be overlaid between gridlines on the destination image. This allows hand-drawn measurements in the source image to be used as a guide for placement of gridlines in the destination image, where reference points / features may be obscured due to ceilings / roofs / etc.

[0054] Once gridlines have been placed, a transformation is applied to the source image to align individual cells of the source image with the position and size of their corresponding cells in the destination image. The user is shown a preview of the resulting gridline transformation to allow for visual inspection of whether the transformation is correct.

[0055] Referring to Fig. 8, if an isolated area of the image needs transformation without affecting the rest of the image, the user may enter a “nested” gridline transform mode by selecting a particular cell of the grid created by intersecting gridlines. Within this nested mode, the same steps of dragging gridlines over the source / destination image are preferably applied (see Fig. 9), however the gridlines only extend to the limits of the parent cell.

[0056] Fig. 10 shows the results of performing a nested gridline transformation on an image - the upper half of the image is correctly transformed without affecting the lower half. Once the nested transformation is correct, the user may exit the nested mode and continue to transform other parts of the image. Once the hand-drawn image of interest is properly transformed, it is ready for eventual alignment with the first, georeferenced image to obtain an aligned georeferenced image. An exemplary method of georeferencing an ungeoreferenced image is set forth, with particular steps shown in Figs. 11 -17 described later.

[0057] First, the image of interest is positioned, or laid over the base outdoor image to facilitate aligning and matching. This initial positioning will not be geospatially correct (as the image of interest lacks any geotags). To facilitate the positioning / overlaying, the image of interest should have a degree of transparency. The degree of transparency may be altered as desired to facilitate better visualization of the image of interest over the base outdoor image.

[0058] After the initial positioning / overlaying of the image of interest on top of the outdoor image, the rest of the georeferencing process is carried out by digitally transforming the image of interest (sufficiently transparent) over the base outdoor image. The use of transparency allows for simple visual validation of results, as the georeferencing process is considered complete when the outlines and features of the image of interest are seen to match the corresponding outlines and features present in the outdoor image.

[0059] Once the image of interest has been transformed to remove any distortions, it can then be rotated using transform controls, the details of which would be appreciated by those of ordinary skill in the art. Due to the earlier step of rotating the outdoor image to its “natural rotation”, it is possible that the image of interest is already correctly aligned / rotated due to the original presentation of the image of interest aligning with the “natural rotation” of the outdoor imagery. In the case of indoor mapping, floorplans are typically drawn in an orientation that maximises the number of horizontal and vertical lines, which is the same orientation as the “natural rotation” described previously. In that situation, little or no alignment or rotation of the image of interest is needed.

[0060] After being correctly rotated, the image of interest is rescaled to dimensionally coincide with the base outdoor image. This scaling transformation can be performed by three different operations. The first is to use transformation controls or touchscreen interactions to visually rescale the image, the details of which would be appreciated by those of ordinary skill in the art. Another operation uses two reference points placed on both the outdoor image and the image of interest, allowing the image of interest to be rescaled so that the reference points in both images align. This operation also allows for repositioning and rotating in a single step for convenience, however the issue of accuracy described above means that individual position / rotation transformations may still be required. The third scaling operation is to rescale the image of interest based on a measurement. This operation is applicable to images of interest that include measurements, such as architectural drawings and images including a scale bar. When performing this operation, a line is drawn over the image of interest to indicate a particular measurement (e.g., the length of a wall), and the corresponding value of that measurement (e.g., 28.0m) is input into a text box. The image of interest is then rescaled to make the measurement visually accurate.

[0061] The final operation is typically a positioning step using a “drag and drop” type method to align outlines and features of the image of interest with the corresponding outlines and features of the outdoor image, the details of which would be appreciated by those of ordinary skill in the art.

[0062] Once the base outdoor image and image of interest are aligned, an inverse transformation may be applied to convert the now-aligned image from a two- dimensional map image (e.g., X and Y pixel coordinates) to an image in a PCS, and a further transformation to convert the aligned image from PCS to GCS. These transformations may use the same or similar mathematical algorithms used to transform the first image described above, as would be appreciated by a person of ordinary skill in the art. The aligned image is then considered georeferenced.

[0063] An example of the method above is now described, with reference to Figs. 1 1 to 17. Fig. 11 shows a preferred embodiment of a method 200 for georeferencing images, using “points of interest.” Referring to Fig. 11 , in step 202, a point of interest (POI) is located on a map image. Next, at step 204, map imagery is selected that provides the best view of the POI (see, e.g., Fig. 2). At step 206, it is determined whether the map is sufficiently rotated to match the “natural rotation” of the POI. If there is insufficient rotation, then in step 208, a user may manually rotate the image on a touch screen, and / or click one or more options to fine-tune rotation with incremental adjustments. The user may click and drag to draw a line along a feature that should be, for example, horizontal (see Fig. 13). Alternatively, the user may type in an exact rotation angle. Once one or more of these side-steps are performed (see step 208 in Fig. 11 ), it is then again determined in step 206 whether the map is sufficiently rotated to match the “natural rotation” of the POI.

[0064] In step 210, a semi-transparent image with the POI is overlaid the base outdoor map image, with arbitrary position / scale / rotation. Then, in step 212, it is determined whether the image / drawing is aligned with the POI’s footprint on the map. If it is, then the image is considered georeferenced at step 214. If the image is not sufficiently aligned, then at step 216, one or more operations are performed.

[0065] For example, at sub-step 218, the image may be scaled using transformation controls (see Fig. 14). At step 220, the image may also be rotated using the transformation controls. The image may be moved by dragging and dropping it at step 222. At step 224, the image may be rescaled based on a particular measurement, such as 28.00m (see Fig. 15). At step 226, the image may be rescaled using two reference points. Referring to Fig. 16, reference points of a line drawn in the image of interest are compared with reference points of a line drawn in the base outdoor image, and the reference points are compared, and used to position the image of interest over the base outdoor image with correct scaling.

[0066] Reverting to Fig. 11 , at step 228, the printed document with the image may be straightened. Fig. 17 shows a method of digitally straightening a document originally portrayed at an angle. It will be appreciated that one or more of the above sub-steps may be omitted, or in a different order as appropriate for a particular situation. For example, steps 220 and 218 above may be performed in a reversed order.

[0067] The foregoing description is by way of example only, and may be varied considerably without departing from the scope of the present disclosure. For example, the second image may be any one of an indoor image that includes a building plan (hand-drawn or formal), or even another outdoor image, with for example, different buildings or outdoor features (e.g., a fountain, or car park). If desired, the image of interest may form the underlying or base image with the outdoor image forming the image overlying the image of interest.

[0068] The method may be enhanced with features to enhance accuracy of building plans that can be used as part of the method. For example, a step of creating a virtual mesh may be included to more efficiently and accurately include internal walls within a building plan. A virtual mesh is created at each intersection of walls by creating a virtual centre line for each wall. Where the centrelines intersect, one or more nodes are created around the intersection to eliminate extraneous lines around the intersection. A virtual mesh can even be created for non-linear (e.g., curved) lines to extrapolate curvatures. Principles of artificial intelligence (Al) may be integrated into one or more aspects of the method described above. For example, Al principles may be used to a chart a maximally efficient or expedient exit path within the building plan. An Al determination may be promulgated by assembling at least one feature vector incorporating physical biological characteristics of a plurality of occupants assigned to at least a portion of the building, and a feature vector of physical characteristics of the building, including exit points and internal distances within the building.

[0069] Concepts of assembling feature vectors would be understood by those of ordinary skill in the Al field, and for simplicity, are not repeated herein. One or more appropriate classifiers may be used to facilitate the Al determination, for example, using a neural network in combination with a Bayesian classifier.

[0070] Al principles may also be integrated for edge detection of imagery (e.g., identifying straight lines in nearby imagery); identifying a north indicator on an image of interest, and then automatically rotating the image of interest to match an outdoor map; and rescaling, positioning and / or rotating involving performing straight line detection, and searching for line pairs with similar portions. An example of rescaling may involve a line being identified in one image with length X and another line being placed in a particular position relative to it with length X / 2. One can then search for similarly positioned and proportioned lines in the other image. Once a match is identified, the image of interest may be rescaled to make both line pairs have identical lengths.

[0071] The present disclosure in a preferred form provides the advantages of more p reci se / accu rate georeferencing since the separate and independent nature of performing steps of positioning, scaling and alignment permit effective consideration of factors associated with a particular environment (e.g., indoor map environments with scaling). Incorporation of Al principles is also facilitated using the separate and independent nature of performing steps of positioning, scaling and alignment. The method of aligning and georeferencing two images, such as described above, is applicable to a broader set of input imagery compared to methods relying on vector data. Two plans may be aligned, even after quality is reduced. The exemplary method described above also allows for the preservation of rotational and scaling accuracy in cases where it is usually impossible to positional accurately (due to roof overhangs, etc.).

[0072] Although methods described herein are well-suited for raster images, aspects of the method(s) may be used to georeferenced any image of interest to an outdoor image.

[0073] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of forms of the embodiments disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims

What is claimed is:1 . A method of georeferencing two raster images with one another, comprising: providing a georeferenced outdoor image and an ungeoreferenced image of interest, the image of interest being at least in part transparent; transforming geographical coordinates associated with the outdoor image into two-dimensional coordinates suitable for presentation on a screen display; independently and separately: positioning the image of interest over the outdoor image; scaling the image of interest to dimensionally coincide with the outdoor image; and aligning the image of interest with the outdoor image to create an aligned image; and inversely transforming the aligned image to geographical coordinates to georeference the aligned image.

2. The method of claim 1 , wherein the step of aligning includes rotating the image of interest.

3. The method of either claim 1 or 2, further comprising, prior to positioning the image of interest, transforming the outdoor image from a geographical coordinate system to a projected coordinate system.

4. The method of claim 3, further comprising, prior to inversely transforming the aligned image, applying an inverse transformation to convert the aligned image from a two-dimensional map image to the outdoor image associated with a projected coordinate system.

5. The method of any one of the above claims, further comprising creating a virtual center line within a wall shown in an internal building plan, and creating a virtual mesh at each intersection of virtual center lines.

6. The method of claim 5, wherein the virtual center line created is non-linear.

7. The method of any one of the above claims, further comprising aligning the image of interest over the outdoor image by dragging the image of interest over the outdoor image, and dropping the image of interest into place.

8. The method of any one of the above claims, wherein image of interest is an internal building plan.

9. The method of any one of claims 1 to 7, wherein the image of interest is another outdoor image.

10. The method of any one of the above claims, further comprising determining an expedient exit path using principles of artificial intelligence, including assembling at least one feature vector incorporating physical biological characteristics of a plurality of occupants assigned to at least a portion of the building, and a feature vector of physical characteristics of the building, including exit points and internal distances within the building.1 1 . The method of any one of the above claims, further comprising applying a gridline transformation to the image of interest to align individual cells of the image of interest with corresponding cells in an aligned image.

12. A method of performing a nested gridline transformation of a portion of a cell of a raster image without affecting another portion of the raster image, comprising: selecting a portion of the raster image; differentially positioning and scaling the portion of the raster image compared to a remainder of the raster image; and aligning the differentially positioned and scaled portion with the remainder of the raster image.

Images