Vision-only gesture relocation

Abstract

Systems and methods for vision-only gesture relocation include an image acquisition device located on an object, the image acquisition device acquiring a query frame of an environment containing the object; and a memory device that stores an image database. Further, the system may include a processor that executes computer readable instructions that direct the processor to execute a coarse matching algorithm to identify a set of coarsely matched data frames stored in the image database that coarsely match the query frame; executing a fine matching algorithm to identify candidate images in the set of roughly matched frames that match the query frame; specifying the candidate image as a matching image based on whether the candidate image satisfies a validity check; and executing a posture solving algorithm based on the acquired query frame, the matching image and parameters for the image acquisition device to estimate the posture of the object in six degrees of freedom.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit and priority of U.S. Patent Application No. 18 / 471,807, filed on September 21, 2023, entitled “Vision-Only Pose Relocalization,” which claims the benefit and priority of Indian Patent Application No. 202311042129, filed on June 23, 2023, entitled “Vision-Only Pose Relocalization,” both of which are incorporated herein by reference in their entirety. Background Art

[0003] Many vehicles are built with navigation systems that attempt to determine the vehicle's position and attitude. Typically, the navigation system obtains information about the vehicle's position and attitude from one or more sensors or receivers mounted to the vehicle. The navigation system receives navigation data about the vehicle's position or environment, which the navigation system uses to determine the vehicle's position and attitude. For example, a navigation system may include inertial sensors and global navigation satellite system receivers, as well as all other sensors that provide or receive information. In addition, some of the sensors used by the navigation system may be image sensors. Summary of the invention

[0004] A system and method for visual-only posture relocalization includes an image acquisition device mounted to an object, the image acquisition device configured to acquire a query frame of an environment containing the object. The system may also include a memory device configured to store an image database. In addition, the system may also include at least one processor, the at least one processor configured to execute computer-readable instructions, the computer-readable instructions directing the at least one processor to execute a coarse matching algorithm to identify a set of roughly matched data frames stored in the image database that roughly match the query frame. In addition, the computer-readable instructions direct the at least one processor to execute a fine matching algorithm to identify a candidate image in the set of roughly matched frames that matches the query frame. Moreover, the computer-readable instructions direct the at least one processor to designate a candidate image as a matching image based on whether the candidate image satisfies a validity check. In addition, the computer-readable instructions direct the at least one processor to execute a posture solving algorithm based on the acquired query frame, the matching image, and parameters for the image acquisition device to estimate the posture of the object in six degrees of freedom. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The accompanying drawings accompany this specification and depict only some embodiments associated with the scope of the appended claims. Therefore, the described and depicted embodiments should not be considered as limiting the scope. The drawings and description describe exemplary embodiments and features thereof with additional specificity and detail, in which:

[0006] Figure 1 is a block diagram illustrating a system for performing vision-only gesture retargeting according to aspects of the present disclosure;

[0007] Figure 2 is a block diagram illustrating a system for preparing data for storage in an image database according to aspects of the present disclosure;

[0008] Figure 3 is a flow chart illustrating a method for performing vision-only gesture relocalization according to aspects of the present disclosure;

[0009] Figure 4 is a block diagram illustrating identification of a set of roughly matching images according to aspects of the present disclosure;

[0010] Figure 5 is a block diagram illustrating identification of a set of exact matching images according to aspects of the present disclosure;

[0011] Fig. 6A and Figure 6B is a diagram illustrating application of a homography matrix when performing a validity check according to aspects of the present disclosure; and

[0012] Figure 7 is a flow chart illustrating a method for performing vision-only gesture relocalization according to aspects of the present disclosure.

[0013] As is common practice, the drawings do not show the various described features to scale, but the features are shown to emphasize the feature's relevance to the exemplary embodiments. DETAILED DESCRIPTION

[0014] The following detailed description refers to the accompanying drawings which form a part of this specification. The drawings show by way of illustration specific illustrative embodiments. However, it is to be understood that other embodiments may be used and that logical, mechanical and electrical changes may be made.

[0015] Systems and methods for visual-only posture relocation are described herein. For example, a navigation system attached to an object includes an image acquisition device that acquires a query frame representation of an environment. In addition, the navigation system may include a database of images, wherein candidate images in the database may represent the environment represented in the acquired query frame. With the database of images and the acquired query frame, the navigation system may execute a series of algorithms to identify candidate matching images (matching candidates) that describe the same environment represented by the information in the acquired query frame. In particular, the navigation system executes a coarse matching algorithm to identify a set of coarse matching candidates in the database of images. When the navigation system has identified a coarse matching candidate, the navigation system executes a fine matching algorithm to identify one or more fine matching candidates. From the fine matching candidates, the navigation system identifies matching candidates that meet the validity check. After identifying the matching image, the navigation system executes a posture solving algorithm using the acquired query frame, the identified matching candidates, and the image acquisition device parameters. The posture solving algorithm allows the navigation system to determine the position of the object in six degrees of freedom using the data acquired from the image acquisition device.

[0016] As described herein, a visual-only posture relocalization system can provide a solution to a problem known as the lost robot problem. The lost robot problem generally refers to a robot losing its position or becoming unorientable in an environment. As vehicles (and objects such as robots) become more and more autonomous, navigation systems are needed to help keep the vehicle properly oriented within the environment. One source of information that can provide positioning information in all six degrees of freedom is image data. In particular, the navigation system acquires image data from the current environment and compares the acquired image data with stored image data representing a known position to determine navigation information. Therefore, the navigation system can acquire image information and determine its posture in six degrees of freedom (latitude, longitude, altitude, yaw, pitch, and roll) without using information acquired from a global navigation satellite system, inertial sensors, or other navigation information sources. In addition, the determined posture can be used as an input to help improve the accuracy of the navigation solution based on the fusion of information from multiple sensors.

[0017] In certain embodiments, the navigation system may perform a repositioning technique called landmark navigation, which uses computer vision to determine the posture of any object attached to the navigation system. As stated, the navigation system may access a database of images, each of which describes a particular area. As part of providing a database of images to the navigation system, raw image data may be prepared to facilitate comparison of stored images with acquired query frames. For example, the raw image data may include satellite or other image data (i.e., laser radar) that may be acquired offline or independently of vehicle operation. For example, when using satellite image data, data representing an area through which an aircraft may fly may be acquired. A processor (such as a server) or multiple processors (such as those employed in massively parallel processing) may use a database of multiple scaled and overlapped preprocessed images of raw image data, wherein the raw image data is provided as input to one or more machine learning models. The machine learning model may provide different descriptors to different images in the image database. The descriptor may then be associated with the image and stored in a database of images together with the image. The database is then stored in a memory device accessible to the navigation system during operation.

[0018] In further embodiments, the machine learning models may generate a different set of descriptors for each image in a database of images, wherein each set of descriptors facilitates comparison using a different comparison algorithm. For example, a first machine learning model may generate a set of coarse descriptors for each image that facilitates a quick comparison of the database image with a query frame acquired by a navigation system. Moreover, a second machine learning model may generate a set of fine descriptors for each image that facilitates a detailed comparison of the database image with a query frame acquired by a navigation system. For example, the coarse descriptors may be global descriptors of the image, and the fine descriptors may be local descriptors of key points within the image.

[0019] In an exemplary embodiment, after acquiring the query frame, the navigation system may immediately calculate the global descriptor of the query frame using a machine learning model or other algorithm. The navigation system may perform a coarse comparison that identifies multiple images in a database having global descriptors that are closest to the global descriptor of the query frame. For example, the navigation system may identify N images in the database having the closest global descriptors. After identifying the N images, the navigation system may calculate the fine descriptors of the query frame using a machine learning model or other algorithm. For example, the machine learning model may identify key points in the query frame. After identifying the fine descriptors of the query frame, the navigation system may compare the fine descriptors found in the query frame with the fine descriptors previously calculated and associated with the closest image identified during the coarse comparison, wherein the navigation system identifies a candidate image having a fine descriptor that best matches the fine descriptor of the query frame.

[0020] In some embodiments, when the navigation system has identified a candidate image, the navigation system can perform a validity check on the candidate image. In particular, the navigation system can use information from the fine match to calculate a homography that maps the footprint of the query frame onto the candidate image. After mapping the footprint onto the candidate image, the navigation system applies the homography to the identified points in the query frame and verifies that the homography maps the identified points to locations within the footprint mapped on the candidate image. If the identified point falls outside the footprint, the navigation system selects a new candidate image that has the next best match to the fine descriptor of the query frame and repeats the homography validity check.

[0021] In certain embodiments, when a candidate image passes a validity check, the navigation system may designate the candidate image as a matching image. The navigation system uses the matching image to calculate the pose of an object associated with the navigation system in up to six degrees of freedom. For example, the navigation system may store parameters for an image acquisition device that describe the relationship between the position of the image acquisition device and the image acquired by the image acquisition device to the object. Using the stored parameters and information describing matching points identified in the query frame and the candidate image, the navigation system may calculate the pose of the object in six degrees of freedom. For example, the navigation system may use the data as input to a perspective N-point (PNP) algorithm that calculates the position (i.e., latitude, longitude, altitude) and attitude (i.e., roll, pitch, yaw) of an object. Therefore, the systems and methods described herein solve the lost robot problem and provide navigation assistance using data acquired by an image acquisition device.

[0022] Figure 1 1 is a block diagram illustrating a potential navigation system 100 that can provide posture relocalization using visual data. For example, the navigation system 100 includes an image sensor 101 (also referred to herein as an image acquisition device), one or more processors 103, and a memory 105. In addition, the memory 105 can store an image database 107, a coarse model 109, and a fine model 111. In some alternative embodiments, the navigation system 100 can include additional sensors 113.

[0023] In certain embodiments, the navigation system 100 can be securely mounted to the object with a known positional relationship. Therefore, the position and attitude of the object can be determined by determining the position and attitude of the navigation system 100. The object can be any object that is operable or moving. For example, the object can be a vehicle, such as an aircraft, a train, a car, a ship, a spacecraft, etc. When the object is a vehicle, the vehicle can be manned or unmanned. In addition, the object can be a personal electronic device or other object. The navigation system 100 can provide information related to the posture of the object. For example, the navigation system 100 can determine the posture of the object relative to a specific reference system in up to six degrees of freedom. When the reference system is the earth reference system, the navigation system 100 can determine the latitude, longitude and altitude of the object. In addition, in the case of being applicable to the object, the navigation system 100 can determine the attitude of the object, such as yaw, pitch and roll.

[0024] In further embodiments, the image sensor 101 may be a device that acquires image data from the environment surrounding the navigation system 100. For example, the image sensor 101 may be a camera, a depth camera, a laser radar, or other image acquisition device. The image sensor 101 may acquire image data as a query frame of image data. The image sensor 101 may be coupled to one or more processors 103, wherein the image sensor 101 provides the acquired image data to the one or more processors 103.

[0025] In an exemplary embodiment, the processor 103 receives acquired image data (query frame) from the image sensor 101 and executes instructions that cause the processor 103 to determine the posture of the object in up to six degrees of freedom. The processor 103 can be a computing device (such as a general-purpose processor, a multi-core processor, a plurality of processors, a dedicated circuit system, etc.). The functions performed by the processor 103 can be implemented using software, firmware, hardware, or any appropriate combination thereof. The processor 103 and other computing devices can be supplemented or incorporated by a specially designed application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processor 103 and other computing devices may also include or run with software programs, firmware or other computer-readable instructions to perform various processing tasks, calculations, and control functions used in the present method and system.

[0026] The method can be implemented by computer executable instructions (such as program modules or components) executed by processor 103 or other computing devices. Typically, program modules include routines, programs, objects, data components, data structures, algorithms, etc. that perform specific tasks or implement specific abstract data types.

[0027] The memory may be a suitable computer-readable storage medium, including, for example, a nonvolatile memory device, including a semiconductor memory device, such as a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), or a flash memory device; a disk, such as an internal hard disk and a removable disk; an optical storage device, such as a compact disk (CD), a digital versatile disk (DVD), a Blu-ray disk, or other media that can carry or store the desired program code as computer-executable instructions or data structures.

[0028] In some embodiments, the processor 103 may be coupled to a memory 105, wherein the memory 105 stores information used by the processor 103 to determine posture information from the acquired image data from the image sensor 101. For example, in addition to computer-readable instructions, the memory 105 also stores an image database 107. The image database 107 is a database of images received from a central repository of image data. The image database may include image data of the environment through which the navigation system 100 travels. For example, when the object is an aircraft, the image data may include overlapping, tiled satellite images of the earth's surface on which the aircraft may travel. In addition, the overlapping, tiled satellite images may be enhanced with three-dimensional information. In further embodiments, for each image in the image database 107, the image database 107 may also include pre-computed coarse and fine descriptors. For example, for coarse descriptors, the image database 107 may store a general descriptor for each image. For fine descriptors, the image database 107 may store pre-computed local descriptors describing features found in each image. In some specific implementations, the image data may be three-dimensional lidar data or other types of image data.

[0029] In other embodiments, the navigation system 100 obtains the information in the image database 107 from a central repository such as an operations center, a data center, or other location. The navigation system 100 may obtain the information in the image database 107 through a wireless or wired communication channel with the central repository. Alternatively, when the navigation system 100 is remotely located from the central repository, the image data may be transmitted to another computer near the navigation system 100 using a computer-readable medium or through a wireless or wired communication channel to be installed on the navigation system 100.

[0030] In certain embodiments, during operation, the image sensor 101 acquires a frame of image data (referred to as a "query frame") and provides the query frame to the processor 103. Upon receiving the query frame, the processor 103 immediately prepares the query frame for comparison with the image data in the image database 107. To prepare the query frame for comparison, the processor 103 may execute the coarse model 109 to determine a coarse descriptor for the query frame, and the processor 103 may execute the fine model 111 to determine a fine descriptor for the query frame. The coarse model 109 and the fine model 111 may be machine learning models that calculate descriptors comparable to the coarse and fine descriptors of the image data stored in the image database 107. In some implementations, the coarse model 109 and the fine model 111 may be the same models used in creating the pre-computed coarse and fine descriptors stored in the image database 107. Alternatively, the coarse model 109 and the fine model 111 may be different models that produce descriptors comparable to the coarse and fine descriptors stored in the image database 107.

[0031] In some embodiments, when the navigation system 100 acquires image data, the navigation system 100 acquires the coarse model 109 and the fine model 111 from a central repository that provides the image. Alternatively, when the coarse model 109 and the fine model 111 are substantially static, the coarse model 109 and the fine model 111 may be loaded onto the memory 105 prior to the initial operation of the navigation system 100. When the coarse model 109 and the fine model 111 need to be updated or reinstalled in the memory 105, the navigation system 100 acquires the coarse model 109 and the fine model 111 through a wireless or wired communication channel with the central repository. Alternatively, when the navigation system 100 is remotely located from the central repository, the coarse model 109 and the fine model 111 may be transmitted to another computer close to the navigation system 100 by using a computer-readable medium or through a wireless or wired communication channel to be installed on the navigation system 100.

[0032] In some embodiments, when the processor 103 determines a coarse descriptor for the query frame by executing the coarse model 109 using the query frame as input, the processor 103 may compare the coarse descriptor with coarse descriptors of different images stored in the image database 107. The processor 103 then identifies a subset of images in the image database 107 that most closely matches the image database 107 based on the comparison of the coarse descriptors.

[0033] In an exemplary implementation, the image database 107 may include pre-processed satellite image tiles, such as United States Geographic Survey (USGS) satellite images. The pre-processed satellite images may be tiles encoded into column vector image descriptors using a global descriptor technique. For example, in one technique for determining a global descriptor, a machine learning model receives an image data frame as input. An example of a machine learning model may be created using a residual neural network (ResNet), where a generalized average pooling (GeM) algorithm may be performed on the last layer of the ResNet, where the GeM is performed with the goal of maximizing average accuracy. The coarse model 109 may be similar to the model used to generate the coarse descriptors of the images in the image database 107. Other algorithms for generating global descriptors, such as a triplet loss method or other algorithms, may also be used.

[0034] In some embodiments, the processor 103 may identify a subset of images by comparing the coarse descriptors of the query frame with the coarse descriptors of the images in the database. For example, the processor 103 may take the dot product of the global descriptor of the query frame with the global descriptors of each image or subset of images in the image database 107. Based on the results of the dot product, the processor 103 may identify a plurality of coarse matching images in the image database 107 that best match the query frame. For example, the processor 103 may identify twenty (or other number) of the best matching images based on the dot product of the coarse descriptors. In some embodiments, the number of coarse matching images in the image database 107 that are identified as the best matches may be adjustable. For example, a user may communicate with the processor 103 via a user interface (not shown) to control the number of images selected. Alternatively, the number of images selected from the image database 107 may be preconfigured or controlled by another external system. By identifying a group of coarse matching images, non-matching images may be excluded from additional processing, thereby increasing processing speed because the group of coarse matching images is processed compared to processing all (or a large number) of images in the image database 107.

[0035] In certain embodiments, when the processor 103 has selected a roughly matched image, the processor 103 may execute a fine matching algorithm to identify a candidate image from the roughly matched image. In order to identify the candidate image, the processor 103 may provide the query frame to the fine model 111. The fine model 111 may be an algorithm that identifies features in an image suitable for comparison with features similarly identified in other images. As described herein, the description of the identified features of the image generated by the fine model 111 may be described as a local descriptor of the image. Therefore, the fine model 111 may generate a local descriptor of the query frame acquired by the image sensor 101. In order to facilitate the comparison between the local descriptors of the image in the image database 107 and the query frame, the image database 107 may also store the local descriptors of the images stored in the image database 107. Although the local descriptors for the images in the image database 107 may be calculated by the fine model 111, a model similar to the fine model 111 may be executed offline by another system to calculate the local descriptors of the images in the image database 107. The pre-computed local descriptors may then be loaded into the image database 107 , where the pre-computed local descriptors may be accessed by the processor 103 for comparison with the acquired query frame.

[0036] In some embodiments, to calculate the local descriptors, the processor 103 executing the fine model 111 may execute a learning-based local descriptor recognition algorithm. In addition, to determine whether the roughly matched image matches the query frame, the fine model 111 executed by the processor 103 may execute an attention graph neural network plus an optimal matching layer. In the execution of the fine model 111, the processor 103 may execute a learning-based local descriptor algorithm on the query frame. The learning-based local descriptor algorithm is a deep learning-based algorithm that extracts local descriptors (key points and descriptors) from an image. The extracted local descriptors are then compared to other similarly processed images. After obtaining the local descriptors, the fine model 111 may instruct the processor 103 to execute a graph neural network algorithm using the extracted local descriptors of the query frame and the stored local descriptors of each roughly matched image. In addition, the fine model 111 may be a graph neural network with an optimal matching layer. The processor 103 may execute the graph neural network algorithm separately for each roughly matched image. In addition, the graph neural network algorithm may also be executed for different orientations and sizes of different roughly matched images to account for different orientations of objects relative to the environment and different distances from surfaces in the environment. The graph neural network algorithm is also a deep learning based algorithm that performs feature matching and geometric verification. The graph neural network algorithm obtains local descriptors from different images and matches the local descriptors using a graph neural network. The graph neural network algorithm generates a confidence measure for the assumed match. The processor 103 can add the generated confidence measures to identify candidate images from the roughly matched images. As used herein, the term candidate image refers to an image in the image database 107 that is identified as the image that most closely represents the environment captured in the query frame.

[0037] In certain embodiments, when the processor 103 has identified a candidate image, the processor 103 may perform a validity check to verify that the candidate image is not identified based on the association of the erroneous data. For example, the processor 103 may match the local descriptors of the query frame to the local descriptors of the candidate image. Based on the matching of the local descriptors, the processor 103 may estimate the homography matrix. Using the homography matrix, the processor 103 may map the boundaries of the query frame and map the features identified in the query frame to the candidate image. Then, the processor 103 may verify that the features identified in the query frame are mapped to the features within the boundaries of the mapping of the query frame on the candidate image. If the query feature is mapped to the feature outside the boundary of the mapping on the candidate image or the boundary of the mapping represents the boundary of the potential mapping that is impossible on the candidate image (i.e., irregular, messy, etc.), the processor 103 may determine that the candidate image is a wrong match. When determining that the candidate image is a wrong match, the processor 103 immediately selects the next best match of the roughly matched image as the candidate image based on the output of the fine model 111, so the processor 103 performs the validity check again. Processor 103 iteratively traverses the roughly matched images until processor 103 finds an image with the best match based on the output of fine model 111 that also passes the validity check. If processor 103 traverses all roughly matched images and still does not find an image that passes the validity check, processor 103 can identify additional roughly matched images.

[0038] In some embodiments, when performing the validity check, the processor 103 may perform a degraded sample consistency (DEGENSAC) method or other methods to ensure the validity of the candidate image. For example, from the local descriptors of the query frame and the candidate image, the processor 103 may select a set of matching descriptors. The processor 103 may generate a homography matrix from the set of matching descriptors. Using the homography matrix generated from the set of matching descriptors, the processor 103 may test the validity of the transformation hypothesis represented by the homography matrix using the local descriptors of the query frame. If the application of the homography matrix causes some local descriptors to fall outside the expected area of ​​the query frame mapped onto the candidate image, the processor 103 determines that the transformation hypothesis is invalid and the candidate image is also invalid.

[0039] In further embodiments, when the processor 103 determines that the candidate image is valid, the processor 103 may designate the candidate image as a matching image. The processor 103 then uses a pose solving algorithm to calculate the pose of the object. For example, in some embodiments, the query frame may be a two-dimensional image, and the matching image may be a three-dimensional image. Based on the correspondence between the two-dimensional image and the three-dimensional image, a perspective n-point (PnP) algorithm (such as EPnP, SQPnP, and P3P) may be used to determine the pose of the object associated with the two-dimensional image. In particular, when using the PnP algorithm, the processor 103 solves the following problems for matching features:

[0040]

[0041] in is the observed pixel position, is a known property of the image sensor 101 that defines the relationship of the image sensor 101 to an object attached to the navigation system 100. In addition, refers to the three-dimensional coordinates of the feature in the matching image associated with the observed pixel position in the earth coordinate system. Using this data, processor 103 can estimate It is the pose of the object attached to the navigation system 100 in six degrees of freedom. In particular, represents the roll, pitch, and yaw of the object, and represents the position of the object. Thus, the processor 103 can calculate the pose of the object in six degrees of freedom. When the processor 103 determines the pose of the object, the processor 103, another processor on the object, or a user can use the pose to guide the navigation of the attached object.

[0042] In some embodiments, the navigation system 100 may include additional sensors 113. The additional sensors 113 include sensors that provide information that the processor 103 can use to determine heading and altitude. For example, the additional sensors 113 may include an inertial measurement unit, a GPS receiver, a barometer, a magnetometer, an altimeter, etc. The additional sensors provide measurements of the movement of the object or environmental information, which the processor 103 uses to calculate the heading and distance from the surface (e.g., the altitude above the ground). Using the heading and distance enables the processor 103 to identify candidate images more quickly.

[0043] In an exemplary embodiment, knowledge of the heading and distance to the surface can reduce the number of comparisons required between the query frame and images in the image database 107. For example, when the object is an aircraft and the image database contains images of terrain over which the aircraft may be flying, the query frame may capture the terrain at a different scale and orientation than the original device used to capture the images in the image database 107. Therefore, when comparing the query frame to the images in the image database 107, each image may be incrementally scaled up or down and rotated to determine the scale and orientation of the query frame. When the heading and altitude are known from information obtained from the additional sensors 113, the processor 103 can determine the scale and orientation of the query frame so that the processor 103 can select the appropriate scale and orientation of the potentially matching images in the image database 107.

[0044] As described above, the execution of the coarse model 109, the fine model 111, the validity check, and the pose calculation is implemented by the processor 103 on the navigation system 100. However, in some embodiments, the processor 103 may communicate with an external system (such as a central server or other computing facility). In such a specific implementation, the processor 103 may transmit the query frame acquired by the image sensor 101 to the external system that determines the pose of the object using the method described above. The external system may then transmit the calculated pose to the processor 103 for guiding the navigation of the object.

[0045] Figure 2 1 is a block diagram of a system 200 for preparing an image database for use by a system that performs gesture relocalization from image data, such as the navigation system 100. As shown, the system 200 includes an image source 221 and a server 223. The server 223 acquires images from the image source 221 and prepares the image information for storage in the image database 107 on the navigation system 100. To prepare the images for storage in the image database 107, the server 223 may include a memory 225, one or more processors 227, and a transceiver 229.

[0046] In some embodiments, image source 221 can be any image data source captured on the area. For example, the image data can be a satellite map obtained from a third party, such as USGS NAIP or USGS HRO imagery. Alternatively, the image data can be lidar data obtained for a specific area. Server 223 processes the image data to create multiple zoom and overlapping tiles of image data. For example, server 223 can receive USGS satellite image data covering approximately 12,000 square kilometers. Server 223 can process images to create tiles with fifty percent overlap at five zoom levels. Server 223 can save both the images received from image source 221 and the processed image tiles in an image database 231 on memory 225. Memory 225 can operate similarly to the above-mentioned memory 105, except that it stores more and different types of information. Since the processing of images obtained from image source 221 can be a computationally intensive task, server 223 can be a supercomputer with processor 227, which has multiple cores or multiple processors 227. Processor 227 may operate similarly to processor 103 described above, except that it may include more processors and process different types of data.

[0047] In further embodiments, after creating different tiles of image data, processor 227 may calculate vertical information so that the tiles of image data are three-dimensional. For example, when the tiles of image data are geographic images, processor 227 may identify vertical geographic information (terrain height) of the tiles of image data. For example, the processor may determine the terrain height of features in the tiles of image data using terrain height information provided by the USGS Shuttle Tomography (STRM) database or the USGS National 3D Elevation Program (3DEP) database. When the terrain height has been applied to the tiles of image data, processor 227 may store the three-dimensional tiles in image database 231.

[0048] In some embodiments, the processor 227 may also calculate descriptors for tiles of image data stored in the image database 231. Pre-calculating descriptors on the server 223 may reduce the computational load on the navigation system 100 and also increase the speed at which the navigation system 100 may determine the pose of an object. To pre-calculate the descriptors, the processor 227 may perform the above combined Figure 1 In some implementations, the coarse model 109 and the fine model 111 stored in the memory 225 may be the same as the coarse model 109 and the fine model 111 described above. Figure 1 The coarse model 109 and the fine model 111 stored in the memory 105 are not the same.

[0049] As described above, the execution of the coarse model 109 generates a global descriptor for a tile of image data. The processor 227 may store the generated global descriptor in the memory 225 as a global descriptor 233. In addition, the execution of the fine model 111 generates a local descriptor for a tile of image data. The processor 227 may store the generated local descriptor in the memory 225 as a local descriptor 235.

[0050] When the descriptor is generated and the image data is prepared, the processor 227 can transmit the above Figure 1 The image data described in the above is transmitted to the navigation system 100 for storage in the image database 107. For example, the processor 227 can transmit all or a subset of the tiles of the three-dimensional image data to the navigation system 100. Moreover, the processor 227 can transmit the global descriptor 233 and the local descriptor 235 associated with the transmitted image data to the navigation system 100. In some embodiments, the processor 227 can also transmit the coarse model 109 and the fine model 111 to the navigation system 100 for use in calculating the descriptors of the acquired query frame. Thus, the server 223 can provide information that enables the navigation system 100 to determine the posture in six degrees of freedom, as described above in conjunction with Figure 1 described.

[0051] Figure 3 is a flow chart of a method 300 for calculating the posture of an object using acquired image data. The steps of the method 300 are referred to above in this document. Figure 1 The method 300 continues at 301, where an image is obtained from a database. For example, the processor 103 may obtain an image from the image database 107, wherein the image in the image database 107 contains information describing a certain area in three dimensions. In addition, the processor 103 may obtain universal and local descriptors of the image in the image database 107. In addition, the method 300 continues at 303, where a query frame is obtained from an image sensor. For example, the processor 103 may obtain a query frame from the image sensor 101.

[0052] In certain embodiments, when the processor 103 has acquired the query frame and the image from the image database 107. The method 300 continues at 305, where a set of roughly matching images is selected. For example, the processor 103 generates a universal descriptor for the query frame using the coarse model 109, and the processor 103 compares the universal descriptor for the query frame with the universal descriptors for the images in the image database 107. Based on the comparison, the processor 103 identifies a set of roughly matching images. When the processor 103 has identified the set of roughly matching images, the method 300 continues at 307, where a candidate image is selected from the set of roughly matching images. For example, the processor 103 generates a local descriptor for the query frame using the fine model 111, and compares the local descriptor for the query frame with the local descriptors of different images in the set of roughly matching images. Based on the comparison, the processor 103 may select a candidate image.

[0053] In an additional embodiment, when a candidate image has been selected, method 300 proceeds to 309, where a validity check is performed on the candidate image. For example, processor 103 may identify a transformation model (e.g., a homography matrix) that transforms some local descriptors of the query frame into corresponding local descriptors in the candidate image. Using the homography matrix, processor 103 may identify the boundaries of the superposition of the query frame on the candidate image. Processor 103 may perform a validity check by determining whether the homography matrix maps the local descriptors of the query frame to locations on the candidate image within the superposition boundaries. Method 300 then proceeds to 311, where processor 103 determines whether the candidate image passes the validity check. If the candidate image does not pass the validity check, method 300 returns to 307 for selecting a new candidate image from the set of roughly matched images.

[0054] In some embodiments, when the candidate image passes the validity check, the method proceeds to 313, where the pose of the object is calculated. For example, the candidate image is designated as a matching image, and using the corresponding local descriptors for the matching image and the query frame together with the parameters for the image sensor, the processor 103 can use a PnP algorithm to identify the pose of the object. After the pose of the object has been identified, the calculated information can then be used for navigation and other activities that potentially benefit from the identified pose information.

[0055] Figure 4 and Figure 5 are diagrams showing a rough comparison and a fine comparison of a query frame with images in an image database, respectively. In particular, Figure 4 4 is a diagram showing the use of the rough model 109 to identify a set of roughly matching images 451. Figure 5 is a diagram illustrating identification of candidate images using the refined model 111 .

[0056] about Figure 4 , systems (such as Figure 1 The navigation system 100 described in the foregoing may obtain a query frame 441 and a set of database images 443 from an image database. The query frame 441 may be provided as an input to the coarse model 109. The coarse model 109 may provide a query general descriptor 447 for the query frame 441. Additionally, the database images 443 may be provided as inputs to the coarse model 109 individually to generate a database general descriptor 449 for each database image 443. The database images 443 may be provided as inputs to the coarse model 109 while the coarse model 109 is generating the query general descriptors 447. Alternatively, the database general descriptors 449 may be pre-computed by the system or another system.

[0057] When the system obtains the query general descriptor 447 and the database general descriptor 449, the system can compare the query general descriptor 447 with the database general descriptor 449 to identify a set of roughly matching images 451. For example, the system can perform a dot product of the query general descriptor 447 and the database general descriptor 449 and identify the database image 443 with the highest dot product result. The system can designate an adjustable number of database images 443 with the highest dot product results as the set of roughly matching images 451.

[0058] about Figure 5, the system may identify a candidate image that matches the query frame 441 from the set of roughly matched images 451 based on the local descriptors generated by the fine model 111. For example, the system may obtain the query frame 441 and provide the query frame 441 as input to the fine model 111, which identifies the query local descriptor 557. In addition, the system may provide each image in the set of roughly matched images 451 as input to the fine model 111 to identify the database local descriptor 559 for each image in the set of roughly matched images 451. While the fine model 111 is generating the query local descriptor 557, the images in the set of roughly matched images 451 may be provided as input to the fine model 111 to generate the database local descriptors 559. Alternatively, the database local descriptors 559 may be pre-computed by the system or by another system.

[0059] When the system obtains the query local descriptor 557 and the database local descriptor 559, the system can compare the query local descriptor 557 with the database local descriptor 559 to identify the candidate image 561. For example, the system can provide the query local descriptor 557 and the database local descriptor 559 as input to a model that provides a confidence measure for a putative match between an image in the set of roughly matched images 451 and the query frame 441. The system can sum the resulting confidence measures to identify the candidate image 561 from the set of roughly matched images 451, where the candidate image 561 has the highest confidence measure sum. The validity of the candidate image 561 can then be checked, as discussed above.

[0060] Fig. 6A and Figure 6B It is shown that validity checks are applied to candidate images. In particular, Fig. 6A shows applying a homography matrix to a query frame that maps the query frame onto a valid candidate image, and Figure 6B Application of a homography matrix to a query frame mapping the query frame onto an invalid candidate image is shown.

[0061] As stated, Fig. 6A Validity check is shown applied to candidate image 561. When performing the validity check, the system can identify a homography matrix based on at least some of the query local descriptors 557 and database local descriptors 559. Using the identified homography matrix, the system can create a mapping 663 of the boundaries of the query frame 441 onto the candidate image 561. After creating the mapping 663, the system can use the homography matrix to map the query local descriptor 557 onto the candidate image 561. Fig. 6A As shown, when the homography matrix is ​​applied to the query local descriptor 557, the query local descriptor 557 is mapped to the database local descriptor 559 located within the mapping 663. When the mapped database local descriptor 559 is located within the mapping 663, Fig. 6A Candidate image 561 in passes the validity check, and candidate image 561 is designated as a matching image.

[0062] Instead, when creating a mapping similar to 663 from the homography matrix Figure 6B In the mapping 663, some query local descriptors 557 are mapped to database local descriptors 559 that are outside the mapping 663. When the query local descriptor 557 is mapped to a database local descriptor 559 outside the mapping 663, Figure 6B 663. In some implementations, the candidate image 561 may fail the validity check when any query local descriptor 557 maps to a database local descriptor 559 that is outside the mapping 663. However, when a certain number or proportion of the query local descriptors 557 map to database local descriptors 559 that are outside the mapping 663, the candidate image 561 may fail the validity check.

[0063] In addition, the mapping 663 created from the homography matrix may represent an impossible potential mapping 663 on the candidate image 561. For example, the mapping 663 may be an extremely irregular shape or a messy mapping. When the mapping 663 is an impossible mapping, the messy mapping or irregular shape may indicate that the homography matrix obtained from the local descriptor is not a valid mapping. Therefore, the candidate image 561 is a wrong match for the query frame 441.

[0064] Figure 7 7 is a flow chart of a method 600 for performing vision-only pose relocalization. The method 700 continues at 701, where a query frame is acquired from an image sensor mounted to an object. The method 700 further continues at 703, where image data is acquired from an image database. Additionally, the method 700 continues at 705, where a coarse matching algorithm is executed to identify a set of roughly matching data frames stored in the image database that roughly match the query frame. Additionally, the method 707 continues at 707, where a fine matching algorithm is executed to identify a candidate image in the set of roughly matching frames that matches the query frame. Additionally, the method continues at 709, where a candidate image is designated as a matching image based on whether the candidate image satisfies a validity check. Additionally, the method 700 continues at 711, where a pose solution algorithm is executed based on the acquired query frame, the matching image, and parameters for the image sensor to estimate the pose of the object in six degrees of freedom.

[0065] Exemplary embodiments

[0066] Embodiment 1 includes a system comprising: an image acquisition device mounted to an object, the image acquisition device being configured to acquire a query frame of an environment containing the object; a memory device being configured to store an image database; and at least one processor being configured to execute computer-readable instructions, the computer-readable instructions directing the at least one processor to: execute a coarse matching algorithm to identify a set of roughly matching data frames stored in the image database that roughly match the query frame; execute a fine matching algorithm to identify a candidate image in the set of roughly matching frames that matches the query frame; designate the candidate image as a matching image based on whether the candidate image satisfies a validity check; and execute a posture solution algorithm based on the acquired query frame, the matching image, and parameters used for the image acquisition device to estimate the posture of the object in six degrees of freedom.

[0067] Embodiment 2 includes a system according to embodiment 1, wherein the computer-readable instructions that instruct the at least one processor to execute the rough matching algorithm further instruct the at least one processor to: calculate a query general descriptor for the query frame; obtain database general descriptors for multiple frames stored in the image database; for each frame in the multiple frames, compare the database general descriptor with the query general descriptor; and designate a certain number of frames in the multiple frames as the group of roughly matched frames.

[0068] Embodiment 3 includes a system according to embodiment 2, wherein the query universal descriptor of the query frame is calculated using a machine learning model stored on the memory device.

[0069] Embodiment 4 includes the system of embodiment 3, wherein the query general descriptor is computed using generalized average pooling.

[0070] Embodiment 5 includes the system of any one of embodiments 2 to 4, wherein the database general descriptor is stored on the memory device after being received from a central repository, wherein the database general descriptor is computed by a plurality of processors at the central repository.

[0071] Embodiment 6 includes a system according to any one of embodiments 1 to 5, wherein the computer-readable instructions that instruct the at least one processor to execute the fine matching algorithm further instruct the at least one processor to: calculate a query local descriptor for the query frame; obtain database local descriptors for multiple frames stored in the image database; for each frame in the group of roughly matched frames, compare the database local descriptor with the query local descriptor; and identify the candidate image in the group of roughly matched frames.

[0072] Embodiment 7 includes a system according to embodiment 6, wherein the query local descriptor of the query frame is calculated using a machine learning model stored on the memory device.

[0073] Embodiment 8 includes the system of any one of embodiments 6-7, wherein the query local descriptor is calculated using a learning-based local descriptor algorithm.

[0074] Embodiment 9 includes the system of any one of embodiments 6 to 8, wherein the database local descriptor is stored on the memory device after being received from a central repository, wherein the database local descriptor is computed by a plurality of processors at the central repository.

[0075] Embodiment 10 includes the system of any one of embodiments 6 to 9, wherein the database local descriptor is compared to the query local descriptor using an attention graph neural network algorithm.

[0076] Embodiment 11 includes a system according to any one of embodiments 1 to 10, wherein the computer-readable instructions that instruct the at least one processor to designate the candidate image as a matching image further instruct the at least one processor to: calculate a homography matrix based on a correspondence between query local descriptors of the query frame and database local descriptors of the candidate image; map a boundary of the query frame onto the candidate image; and determine that the candidate image is the matching image when at least one of the query local descriptors corresponds to the database local descriptor within the mapped boundary and the mapped boundary is associated with a valid mapping on the candidate image.

[0077] Embodiment 12 includes a system according to any one of embodiments 1 to 11, wherein the posture solving algorithm is a perspective N-point algorithm.

[0078] Embodiment 13 includes a system according to any one of Embodiments 1 to 12, the system further comprising one or more additional sensors, wherein the one or more additional sensors provide navigation measurements of heading and altitude, and wherein the at least one processor performs the coarse matching algorithm on the data in the image database at an orientation associated with the heading and a scale associated with the altitude.

[0079] Embodiment 14 includes a method, which includes: acquiring a query frame from an image sensor mounted to an object; acquiring image data from an image database; executing a coarse matching algorithm to identify a set of roughly matching data frames stored in the image database that roughly match the query frame; executing a fine matching algorithm to identify a candidate image in the set of roughly matching frames that matches the query frame; designating the candidate image as a matching image based on whether the candidate image satisfies a validity check; and executing a posture solving algorithm based on the acquired query frame, the matching image and parameters for the image sensor to estimate the posture of the object in six degrees of freedom.

[0080] Embodiment 15 includes the method according to embodiment 14, wherein executing the rough matching algorithm also includes: calculating the query universal descriptor of the query frame; obtaining the database universal descriptors of multiple frames stored in the image database; for each frame in the multiple frames, comparing the database universal descriptor with the query universal descriptor; and designating a certain number of frames in the multiple frames as the group of roughly matched frames.

[0081] Embodiment 16 includes the method according to embodiment 15, wherein performing the coarse matching algorithm includes calculating the query universal descriptor of the query frame using a machine learning model.

[0082] Embodiment 17 includes a method according to any one of Embodiments 14 to 16, wherein executing the fine matching algorithm further includes: calculating a query local descriptor of the query frame; obtaining database local descriptors of multiple frames stored in the image database; for each frame in the group of roughly matched frames, comparing the database local descriptor with the query local descriptor; and identifying the candidate image in the group of roughly matched frames.

[0083] Embodiment 18 includes the method according to embodiment 17, wherein the query local descriptor of the query frame is calculated using a machine learning model.

[0084] Embodiment 19 includes a method according to any one of embodiments 14 to 18, wherein specifying the matching candidate further includes: calculating a homography matrix based on the correspondence between the query local descriptor of the query frame and the database local descriptor of the candidate image; mapping the boundary of the query frame to the candidate image; and determining that the candidate image is the matching image when the query local descriptor corresponds to the database local descriptor within the mapped boundary and the mapped boundary is associated with a valid mapping on the candidate image.

[0085] Embodiment 20 includes a system comprising: a central repository comprising: a plurality of processors; and an image database storing a repository of image data acquired from a third party; wherein the plurality of processors execute a plurality of machine learning models using a portion of the repository of image data to create a plurality of three-dimensional images, local descriptors and universal descriptors of images in the plurality of three-dimensional images; and a navigation system comprising: an image sensor mounted to an object, the image sensor configured to acquire a query frame of an environment containing the navigation system; a memory device configured to store the three-dimensional image received from the central repository; The invention relates to a method for obtaining a three-dimensional image, a local descriptor, and a general descriptor; and at least one processor configured to execute computer-readable instructions, wherein the computer-readable instructions direct the at least one processor to: execute a coarse matching algorithm to identify a group of roughly matching data frames in the three-dimensional image that roughly match the query frame; execute a fine matching algorithm to identify a candidate image that matches the query frame from a group of roughly matching candidates; designate the candidate image as a matching image based on whether the candidate image satisfies a validity check; and execute a posture solving algorithm based on the acquired query frame, the matching image, and parameters of the image sensor to estimate the posture of the object in six degrees of freedom.

[0086] Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. Therefore, it is manifestly intended that the present invention be limited only by the claims and the equivalents thereof.

Claims

1. A system, comprising: an image acquisition device mounted to the object, the image acquisition device being configured to acquire a query frame of an environment containing the object; a memory device configured to store an image database; and at least one processor configured to execute computer-readable instructions directing the at least one processor to: executing a rough matching algorithm to identify a set of roughly matching data frames stored in the image database that roughly match the query frame; executing a fine matching algorithm to identify candidate images in the set of roughly matched frames that match the query frame; designating the candidate image as a matching image based on whether the candidate image satisfies the validity check; as well as A pose solving algorithm is performed based on the acquired query frame, the matching image, and parameters for the image acquisition device to estimate the pose of the object in six degrees of freedom.

2. The system of claim 1 , wherein the computer readable instructions that direct the at least one processor to designate the candidate image as a matching image further direct the at least one processor to: Calculating a homography matrix based on a correspondence between a query local descriptor of the query frame and a database local descriptor of the candidate image; Mapping the boundary of the query frame onto the candidate image; as well as The candidate image is determined to be the matching image when at least one of the query local descriptors corresponds to the database local descriptor within the mapped boundary and the mapped boundary is associated with a valid mapping on the candidate image.

3. A method comprising: acquiring a query frame from an image sensor mounted to the object; Acquire image data from an image database; executing a rough matching algorithm to identify a set of roughly matching data frames stored in the image database that roughly match the query frame; executing a fine matching algorithm to identify candidate images in the set of roughly matched frames that match the query frame; designating the candidate image as a matching image based on whether the candidate image satisfies the validity check; as well as A pose solution algorithm is performed based on the acquired query frame, the matching image, and parameters for the image sensor to estimate the pose of the object in six degrees of freedom.

Images