Cross-mode image-related surface positioning system

Abstract

A cross-mode image-related surface positioning system is provided that includes an infrared camera for capturing an infrared image. A database stores images in the visible spectrum and location information associated with each stored image. When the infrared camera captures the infrared image, a controller selects a stored image among the stored images in the database. The selection of the stored image of the stored images is based on an estimated position of the vehicle including the infrared camera. The controller converts the selected stored image into a grayscale image. The controller determines an absolute correlation surface between the captured infrared image and the grayscale image. The controller determines a maximum first peak in the determined absolute correlation surface. The controller updates a navigation system of the vehicle based on a peak-related position associated with the determined maximum first peak.

Description

BACKGROUND

[0001] One common system for determining the location of a vehicle is a global navigation satellite system (GNSS) such as a global positioning system (GPS). However, sometimes the GPS is not available or is not reliable because the system is being spoofed. One method of determining a location without GPS is by taking an image of the area in front of or below the vehicle and comparing the image to images stored in an image location database to determine if any of the content in the image matches. If enough of the content in the image matches within a selected certainty, then the location can be determined.

[0002] For vehicles that operate both during the day and at night, images from an infrared spectrum camera can be used. However, the images stored in the image location database are typically taken and stored in the visible spectrum. For example, satellite map reference images taken in the visible spectrum are typically used in the image location database. Comparing different types of images for similar content can cause problems that can result in inaccurate location determinations. The problem comes from the visual differences in the earth's surface when viewed in the thermal infrared versus the visible spectrum. The visual differences present a challenge to the algorithm in relating the two images due to the differences in the content of the two images. For example, there is a strong image inversion between two images of asphalt because an asphalt road appears dark in the visible spectrum, but it can appear bright in the thermal infrared spectrum on a hot day. These visual differences can result in inaccurate comparisons.

[0003] For the above reasons and other reasons that will become apparent to those skilled in the art upon reading and understanding the present specification, there is a need in the art for an effective and efficient way to compare visible images and infrared images. SUMMARY

[0004] The following summary is made by way of example and not limitation. The purpose of the summary is to enable the reader to understand some of the aspects of the described subject matter. The embodiments provide a system for comparing cross-modal images such as infrared spectrum images and visible spectrum images by phase correlation.

[0005] In one embodiment, a cross-modal image correlation surface positioning system is provided. The system includes an infrared camera to capture an infrared image, a database, and a controller. The database stores images in a visible spectrum and location information associated with each stored image in the database. The controller is configured to select a stored image among the stored images in the database when the infrared camera captures the infrared image. The selection of the stored image among the stored images in the database is based on an estimated location of a vehicle that includes the infrared camera. The controller is configured to convert the selected stored image to a grayscale image. The controller is further configured to determine an absolute correlation surface between the captured infrared image from the infrared camera and the grayscale image. The controller is configured to determine a maximum first peak in the determined absolute correlation surface to verify a peak correlation location between an aspect in the captured infrared image from the infrared camera and the grayscale image from the selected stored image. The controller is further configured to update a navigation system of the vehicle based on the peak correlation location associated with the determined maximum first peak.

[0006] In another embodiment, a system including a cross-modal image correlation surface positioning system and a navigation system is provided. The cross-modal image correlation surface positioning system includes an infrared camera to capture an infrared image, a database, and a controller. The database stores images in a visible spectrum and location information associated with each stored image in the database. The controller is configured to select a stored image among the stored images in the database when the infrared camera captures the infrared image. The selection of the stored image among the stored images in the database is based on an estimated location of a vehicle that includes the infrared camera. The controller is configured to convert the selected stored image to a grayscale image. The controller is further configured to determine an absolute correlation surface between the captured infrared image from the infrared camera and the grayscale image. The controller is configured to determine a maximum first peak in the determined absolute correlation surface. The navigation system is in communication with the controller. The navigation system is configured to update a navigation output based on a peak correlation location associated with the determined maximum first peak.

[0007] In another embodiment, a method of using a cross-modal image correlation surface positioning system is provided. The method includes capturing an infrared image with an infrared camera; estimating a location of a vehicle that includes the infrared camera when the infrared image is taken; retrieving a visible spectrum image from a database based on the estimated location; converting the visible spectrum image to a grayscale image; determining an absolute correlation surface between the grayscale image and the infrared image; identifying a maximum first peak in the determined absolute correlation surface; and updating a navigation system of the vehicle using information gathered by the identification of the maximum first peak. BRIEF DESCRIPTION OF DRAWINGS

[0008] The present application can be more easily understood and further advantages and uses thereof will be more readily apparent, when considered along with the following detailed description and the accompanying drawings in which:

[0009] Figure 1 is a block diagram of a vehicle having a cross-modal image-related surface positioning system according to example aspects of the present application;

[0010] Figure 2A illustrates an example infrared camera image;

[0011] Figure 2B illustrates a grayscale conversion of a visible spectrum map according to example aspects of the present application;

[0012] Figure 3A illustrates a translation-related surface between a camera image and a map image according to example aspects of the present application;

[0013] Figure 3B illustrates an absolute value of a translation-related surface between a camera image and a map image according to example aspects of the present application; Figure 3A

[0014] Figure 3C further illustrates applying an absolute value of Figure 3B to a translation-related surface of Figure 3A ; and

[0015] Figure 4 illustrates a cross-modal image-related surface positioning flowchart according to example aspects of the present application.

[0016] In accordance with common practice the various features described with reference to the drawings are not necessarily drawn to scale but to emphasize specific features relevant to the present application. Reference signs are used throughout the drawings and the text to indicate like components. DETAILED DESCRIPTION

[0017] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which are shown by way of illustration specific embodiments in which the application can be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the application, and it is to be understood that other embodiments can be utilized and that changes can be made without departing from the spirit and scope of the present application. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present application is defined only by the claims and equivalents thereof.

[0018] ​Embodiments of the invention provide a system for efficiently comparing cross-modal images such as infrared and visible spectral images by phase correlation. In embodiments, the absolute value of the phase correlation is used to determine an "absolute correlation surface". In one example, at least a maximum peak value is determined from the absolute correlation surface. In an example, the maximum peak value is used to determine a peak correlation location which can be used for navigation purposes as described herein. In another example, a second maximum peak value is used as a quality metric.

[0019] Figure 1 A block diagram of a vehicle having a cross-modal image correlation surface positioning system 101 implemented by the vehicle 100 according to an example aspect of the invention is illustrated. The cross-modal image correlation surface positioning system 101 includes a controller 102, a memory 104, and an infrared (IR) camera 108.

[0020] Generally, the controller 102 can include any one or more of a processor, a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or equivalent discrete or integrated logic circuitry. In some example embodiments, the controller can include multiple components, such as one or more microprocessors, one or more controllers, one or more DSPs, one or more ASICs, one or more FPGAs, and other discrete or integrated logic circuitry in any combination. The functions attributed to the controller 102 herein can be embodied as software, firmware, hardware, or any combination thereof. The controller 102 can be a part of a system controller or a component controller. The memory 104 can include computer readable operational instructions that, when executed by the controller 102, provide the functions of the cross-modal image correlation surface positioning system 101. Such functions can include the functions of determining a correlation surface and an absolute correlation surface described below. The computer readable instructions can be encoded within the memory. The memory is a suitable non-transitory storage medium including any volatile, non-volatile, magnetic, optical, or electrical media, such as but not limited to a random access memory (RAM), read only memory (ROM), non-volatile RAM (NVRAM), electrically erasable programmable ROM (EEPROM), flash memory, or any other memory technology.

[0021] The memory 104 in this example includes an image database 106. The image database 106 includes images associated with location information. While the image database 106 is illustrated as being located in the memory 104, the image database 106 can be in another location. The example images stored in the image database 106 are satellite images in the visible spectrum. As discussed, the database 106 includes location information associated with each image in the database. The location information can include location information associated with features captured within each image in the image database.

[0022] In one example, the controller 102 in communication with the IR camera 108 controls the IR camera 108 to take IR images. In this example, the controller 102 is further in communication with a navigation system 110. The navigation system 110 can include a global navigation satellite system (GNSS), such as a global positioning system (GPS). In addition, the navigation system 110 can include other systems for determining location, such as an inertial navigation system (INS), a star tracker, etc. In another example, the image-based positioning system is the only navigation system. In addition, in one example, at least the controller 102 can be part of the navigation system 110.

[0023] Figure 1 An example of the vehicle control system 112 is further illustrated. The vehicle control system 112 controls the operation of the vehicle 100, including the direction in which the vehicle 100 is traveling. In one example, the vehicle control system 112 controls the vehicle 100 based at least in part on the location determined from the intersection pattern image correlation surface positioning system 101.

[0024] In one example, the controller 102 instructs the IR camera 108 to take an image. Based on the estimated location of the vehicle, the controller retrieves a stored image from the database 106 associated with the estimated location. The estimated location can be determined with another location determination system, such as but not limited to a GPS / INS system, a star tracker, dead reckoning, etc. One example application is where GPS signals in a GPS / INS system are not available or compromised, the estimated location information from the INS can be used to provide an estimated location to identify an associated image in the database 106. In addition, in an avionics example, the use of a flight plan, travel time, and travel rate can be used to estimate a general location that can be used to retrieve an associated image.

[0025] Figure 2A An example of an IR camera image 200 taken by the IR camera 108 is illustrated in the example. As discussed above, the images stored in the database 106 are typically satellite images in the visible spectrum. The images in the database 106 can be converted to grayscale by the controller 102. An example of a visible spectrum image from the database 106 that is converted to a grayscale image isFigure 2B Gray scale image 202 overlaid with the IR camera image 200 of Figure 2A Gray scale image 202 overlaid with the IR camera image 200 of

[0026] Figure 3A An example of a phase correlation process 300 using IR camera image 302 and database image 304 from database 106 that has been converted to gray scale. The result of the phase correlation is a translation correlation surface 306 as illustrated in translation correlation surface map 310. Translation correlation surface 306 of translation correlation surface map 310 includes a positive peak 320 and a negative peak 330 that indicate correlation within IR camera image 302 and database image 304.

[0027] Figure 3B An example of an inverse phase correlation process 340 using IR camera image 302 and inverse database image 342. By inverting the database image and then performing a phase correlation, a translation correlation surface is generated as illustrated in translation correlation surface map 350. By using the inverse database image 342, the negative peak (330 of 302) becomes positive, adding to the existing positive peak (320 of 302), resulting in an increased positive peak 354 that is easier to identify or detect. The effect of using the absolute value to identify peak correlation is further illustrated in correlation difference example 360. As illustrated, when translation correlation surface map 310 includes a positive peak 320 and a negative peak 330, absolute translation correlation surface map 350 includes one large positive peak 354. Absolute correlation surface 352 includes image translation values based on the coordinates of the peaks to locate the position of the feature. Figure 3A Figure 3A Figure 3C

[0028] Referring to Figure 4 A method of implementing a cross-modal image correlation surface localization system is provided in cross-modal image correlation surface localization flowchart 400. Cross-modal image correlation surface localization flowchart 400 is provided in a sequential series of blocks. At least some of these blocks are executed by controller 102 implementing operational instructions stored in memory 104. However, in other examples, the order of the blocks can occur in a different order or in parallel. Accordingly, the present invention is not limited to a particular sequential series of blocks.

[0029] ​​​In the example cross-modal image correlation surface localization flowchart 400, the process begins at block 402 where an IR image is captured. In one example, the IR image is captured with an IR camera under the direction of the controller 102. When an IR image is captured in block 402, the position of the vehicle is estimated at block 404. As described above, the estimated position can be determined using a position determination system that is part of the navigation system 110, such as but not limited to a GPS / INS system, a star tracker, dead reckoning, a path of travel tracking system, etc. The image in the database 106 that is associated with the estimated position determined in block 405 is selected.

[0030] At block 406, the selected image in the database 106 is converted to grayscale. In one example, the selected image can be a satellite image. At block 410, an absolute correlation surface between the current IR image and the grayscale image from the database is determined.

[0031] Once the absolute correlation surface is determined at block 410, a first maximum peak in the absolute correlation surface is identified at block 412. It is determined at block 413 whether the maximum first peak has been found. If the maximum first peak cannot be found, the process continues at block 402 to take another IR image. In one example, once the maximum first peak is found at block 413, an exclusion window is established around the maximum first peak surface at block 414. A second or secondary peak outside of the exclusion window is found or determined at block 416. The secondary peak is less than the maximum first peak surface.

[0032] At block 418, the secondary peak is used for a quality metric to help verify the position determined by the first maximum peak. At block 420, the position of the vehicle 100 is determined based on the maximum first peak verified with the secondary peak. In one example, the peak associated position that results in the maximum first peak in the absolute correlation surface is used to determine the position of the vehicle 100.

[0033] In this example, the determined position is provided to the navigation system at block 422. In one example, the information gathered by identifying the maximum first peak that includes the determined position is used to update the navigation system 110. The navigation system 110 can be updated by the determined position by verifying the estimated position. In another example, the navigation system 110 can be updated by calibrating the navigation system 110. Further, in one example, the navigation system 110 is updated by changing the output position of the navigation system 110 to the determined position.

[0034] Further, in another example, the vehicle is operationally controlled by a vehicle control system 112 based at least in part on the location determined at block 424 by the cross- modality image correlation surface positioning system. As described above, the vehicle control system 112 can control a direction of travel of the vehicle 100. The process then continues at block 402 with another infrared image being taken.

[0035] Example Embodiments

[0036] Example 1 includes a cross-modality image correlation surface positioning system. The system includes an infrared camera to capture an infrared image, a database, and a controller. The database stores images in a visible spectrum and location information associated with each stored image in the database. The controller is configured to select a stored image among the stored images in the database when the infrared camera captures the infrared image. The selection of the stored image among the stored images is based on an estimated location of a vehicle that includes the infrared camera. The controller is configured to convert the selected stored image to a grayscale image. The controller is further configured to determine an absolute correlation surface between the captured infrared image from the infrared camera and the grayscale image. The controller is configured to determine a maximum first peak in the determined absolute correlation surface. The controller is further configured to update a navigation system of the vehicle based on a peak correlation location associated with the determined maximum first peak.

[0037] Example 2 includes the system of Example 1, further comprising a memory in communication with the controller, the memory storing operational instructions implemented by the controller.

[0038] Example 3 includes the system of any one of Examples 1-2, further comprising a vehicle control system configured to control an operation of the vehicle based at least in part on the peak correlation location associated with the determined maximum first peak.

[0039] Example 4 includes the system of any one of Examples 1-3, wherein the update to the navigation system includes at least one of: verifying the estimated location, calibrating the navigation system, and using the peak correlation location in determining a navigation system location.

[0040] Example 5 includes the system of any one of Examples 1-4, wherein the controller is further configured to establish an exclusion window around the peak correlation location in the absolute correlation surface.

[0041] Example 6 includes the system of Example 5, wherein the controller is configured to find a second peak in the absolute correlation surface outside of the exclusion window, and use a second peak location associated with the second peak for a quality metric.

[0042] Example 7 includes a system comprising a cross-modal image correlation surface positioning system and a navigation system. The cross-modal image correlation surface positioning system comprises an infrared camera to capture infrared images, a database, and a controller. The database stores images in a visible spectrum and location information associated with each stored image in the database. The controller is configured to select a stored image in the stored images in the database when the infrared camera captures the infrared images. The selection of the stored image in the stored images is based on an estimated location of a vehicle comprising the infrared camera. The controller is configured to convert the selected stored image to a grayscale image. The controller is further configured to determine an absolute correlation surface between the captured infrared image from the infrared camera and the grayscale image. The controller is configured to determine a maximum first peak in the determined absolute correlation surface. The navigation system is in communication with the controller. The navigation system is configured to update a navigation output based on a peak correlation location associated with the determined maximum first peak.

[0043] Example 8 includes the system of Example 7, wherein the cross-modal image correlation surface positioning system further comprises a memory in communication with the controller. The memory stores operational instructions implemented by the controller.

[0044] Example 9 includes the system of any of Examples 7-8, wherein the peak correlation location associated with the determined maximum first peak is used by the navigation system for at least one of verifying the estimated location and calibrating the navigation system.

[0045] Example 10 includes the system of any of Examples 7-9, wherein the peak correlation location associated with the determined maximum first peak is used by the navigation system to determine a navigation system location.

[0046] Example 11 includes the system of any of Examples 7-10, further comprising a vehicle control system configured to control an operation of the vehicle based at least in part on the peak correlation location associated with the determined maximum first peak.

[0047] Example 12 includes the system of any of Examples 7-11, wherein the controller of the cross-modal image correlation surface positioning system is further configured to establish an exclusion window around the peak correlation location in the absolute correlation surface.

[0048] Example 13 includes the system of Example 12, wherein the controller of the cross- modality image correlation surface positioning system is further configured to find a second peak in the absolute correlation surface outside of the exclusion window and use a second peak location associated with the second peak for a quality metric.

[0049] Example 14 includes a method of using a cross-modality image correlation surface positioning system. The method includes capturing an infrared image with an infrared camera; estimating a position of a vehicle including the infrared camera when the infrared image is taken; retrieving a visible spectrum image from a database based on the estimated position; converting the visible spectrum image to a grayscale image; determining an absolute correlation surface between the grayscale image and the infrared image; identifying a maximum first peak in the determined absolute correlation surface; and updating a navigation system of the vehicle using information gathered by the identification of the maximum first peak.

[0050] Example 15 includes the method of Example 14, further comprising controlling vehicle operations based on the identified maximum first peak.

[0051] Example 16 includes the method of any of Examples 14-15, further comprising establishing an exclusion window around the identified first peak.

[0052] Example 17 includes the method of any of Examples 14-16, further comprising identifying a secondary peak in the absolute correlation surface outside of the exclusion window.

[0053] Example 18 includes the method of Example 17, further comprising using the secondary peak for a quality metric.

[0054] Example 19 includes the method of any of Examples 14-18, wherein the updating of the navigation system includes at least one of verifying the estimated position and calibrating the navigation system.

[0055] Example 20 includes the method of any of Examples 14-19, further wherein the updating of the navigation system includes using a peak correlation location associated with the maximum first peak as a determined navigation system position.

[0056] While specific embodiments have been illustrated and described herein, it will be appreciated that any arrangement calculated to achieve the same purpose can be substituted for the specific embodiments shown. This application is intended to cover any adaptations or variations of the present application. Therefore, it is manifestly intended that this application be limited only by the following claims and equivalents thereof.

Claims

1. A cross-pattern image-related surface localization system (101), the system (101) comprising: An infrared camera (108) is used to capture infrared images; A database (106) that stores images in the visible spectrum and location information associated with each stored image in the database (106); and A controller (102) is configured to select a stored image from the stored images in the database (108) when the infrared camera (108) captures the infrared image, the selection of the stored image from the stored images being based on an estimated position of the vehicle (100) including the infrared camera (108), the controller (102) being configured to convert the selected stored image into a grayscale image, the controller (102) being further configured to determine an absolute correlation surface between the captured infrared image from the infrared camera (108) and the grayscale image, the controller (102) being configured to determine a maximum first peak value in the determined absolute correlation surface, and the controller (102) being further configured to update the navigation system (112) of the vehicle (100) based on the peak correlation position associated with the determined maximum first peak value.

2. A system comprising: A cross-mode image-correlated surface localization system (101), the cross-mode image-correlated surface localization system comprising, Infrared camera (108), the infrared camera being used to capture infrared images, Database (106), which stores images in the visible spectrum and location information associated with each stored image in the database (106), A controller (102) configured to select a stored image from the stored images in the database (106) when the infrared camera (108) captures the infrared image, the selection of the stored image from the stored images being based on an estimated position of the vehicle (100) including the infrared camera (108), the controller (102) being configured to convert the selected stored image into a grayscale image, the controller (102) being further configured to determine an absolute correlation surface between the infrared image captured from the infrared camera (108) and the grayscale image, the controller (102) being configured to determine a maximum first peak value in the determined absolute correlation surface; and A navigation system (110) that communicates with the controller (102) is configured to update navigation output based on peak-related positions associated with a determined maximum first peak value.

3. A method for locating a surface using a cross-pattern image correlation system, the method comprising: Infrared images were captured using an infrared camera (108); When the infrared image is captured, the location of the vehicle (100) including the infrared camera (108) is estimated; Retrieve visible spectral images from database (106) based on the estimated location; Convert the visible spectrum image into a grayscale image; Determine the absolute correlation surface between the grayscale image and the infrared image; The largest first peak value in the absolutely correlated surface identified by the identifier; as well as The navigation system (110) of the vehicle (100) is updated using information collected by identifying the maximum first peak value.

Images