Unified visual positioning architecture

Abstract

Systems and methods for providing a unified visual positioning architecture are described herein. In some implementations, a system includes an image acquisition device mounted to an object, the image acquisition device configured to acquire a query frame of an environment containing the object. The system also includes a memory device configured to store an image database. Further, the system includes at least one processor configured to execute computer readable instructions that direct the at least one processor to identify a set of data in the image database that potentially matches the query frame; identifying a visual localization normal form in the plurality of visual localization normal forms; and determining a pose of the object using the set of data, the query frame, and a lens characteristic of the image acquisition device as inputs to the visual localization normal form.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Indian Provisional Application No. 202311058474, filed on August 31, 2023, entitled “UNIFIED VISUAL LOCALIZATION ARCHITECTURE,” the contents of which are incorporated herein in their entirety. Background Art

[0003] Many vehicles are built with navigation systems that attempt to determine the position and attitude of the vehicle. Typically, the navigation system obtains information about the position and attitude of the vehicle from one or more sensors or receivers mounted to the vehicle. The navigation system receives navigation data about the position or environment of the vehicle, and the navigation system uses the navigation data to determine the position and attitude of the vehicle. For example, the navigation system may include inertial sensors and global navigation satellite system receivers, as well as other sensors that provide or receive information. In addition, some of the sensors used by the navigation system may be image sensors. Summary of the invention

[0004] Systems and methods for providing a unified visual localization architecture are described herein. In some specific implementations, a system includes an image acquisition device mounted to an object, the image acquisition device being configured to acquire a query frame of an environment containing the object. The system also includes a memory device configured to store an image database. In addition, the system includes at least one processor configured to execute computer-readable instructions that instruct the at least one processor to identify a set of data in the image database that potentially matches the query frame. In addition, the computer-readable instructions instruct the at least one processor to identify a visual localization paradigm among multiple visual localization paradigms. In addition, the computer-readable instructions instruct the at least one processor to use the set of data, the query frame, and the lens characteristics of the image acquisition device as input to the visual localization paradigm to determine the pose of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The accompanying drawings accompany this specification and depict only some embodiments associated with the scope of the appended claims. Therefore, the described and depicted embodiments should not be considered as limiting the scope. The drawings and description describe exemplary embodiments and features thereof with additional specificity and detail, in which:

[0006] Figure 1 is a block diagram illustrating a system implementing a unified visual positioning architecture according to aspects of the present disclosure;

[0007] Figure 2is a block diagram illustrating a system for preparing data for storage in an image database according to aspects of the present disclosure;

[0008] Figure 3 is a flow chart illustrating a method for performing visual positioning within a unified architecture according to aspects of the present disclosure;

[0009] Figure 4 is a block diagram illustrating a unified visual positioning architecture according to aspects of the present disclosure;

[0010] Figure 5 is a block diagram illustrating a candidate image identifier according to aspects of the present disclosure;

[0011] Figure 6 is a block diagram illustrating a visual localization paradigm for achieving fine matching according to aspects of the present disclosure;

[0012] Fig. 7A and Figure 7B is a diagram illustrating application of a homography matrix when performing validity checks within a visual localization paradigm according to aspects of the present disclosure;

[0013] Figure 8 is a block diagram illustrating a visual localization paradigm using point clouds according to aspects of the present disclosure;

[0014] Fig. 9 is a block diagram illustrating a visual localization paradigm for implementing neural rendering according to aspects of the present disclosure;

[0015] Fig.10 is a block diagram illustrating a visual localization paradigm for implementing learning depth according to aspects of the present disclosure;

[0016] Fig.11 is a block diagram illustrating a visual localization paradigm for achieving pose approximation according to aspects of the present disclosure;

[0017] Fig.12 is a block diagram illustrating a visual localization paradigm for implementing geometric transformations according to aspects of the present disclosure; and

[0018] Fig.13 is a flow chart illustrating a method for performing visual positioning within a unified architecture according to aspects of the present disclosure.

[0019] As is common practice, the drawings do not show the various described features to scale, but the features are shown to emphasize the feature's relevance to the exemplary embodiments. DETAILED DESCRIPTION

[0020] The following detailed description refers to the accompanying drawings which form a part of this specification. The drawings show by way of illustration specific illustrative embodiments. However, it is to be understood that other embodiments may be used and that logical, mechanical and electrical changes may be made.

[0021] Systems for implementing a unified visual positioning architecture and methods for using a unified visual positioning architecture are described herein. For example, a navigation system attached to an object includes an image acquisition device that acquires a query frame representation of an environment. In addition, the navigation system may include a database of images, wherein the images in the database depict scenes representing a potential environment of the object. The navigation system uses the database of images and the acquired query frame to identify a set of potential candidate images in the database of images, wherein the set of potential candidate images potentially matches the data captured in the query frame. After identifying the set of potential candidate images, the system may select and implement one or more visual positioning paradigms based on the operating environment of the object. The system uses the query image and the set of potential candidate images as inputs to the selected visual positioning paradigm to locate objects within the environment.

[0022] The visual positioning system can estimate the camera position based on the acquired image. Typically, the visual positioning system is part of an object having a mounted image capture device, wherein the image capture device generates a query image that depicts a scene representation of the object's environment. Using the acquired query image, the visual positioning system can compare the query image to previously acquired images depicting the environment, wherein the previously acquired images are associated with a known geographic location. Based on the comparison, information describing the characteristics of the image capture device, and the relationship of the image capture device to the object, the visual positioning system can determine pose information for the object.

[0023] When a vision localization system attempts to determine pose information, the vision localization system may select from a number of different vision localization paradigms for identifying the pose of an object. However, different vision localization paradigms are not equivalent to each other. In particular, these paradigms may have different benefits and disadvantages. For example, some paradigms may provide more accurate solutions, while others may execute faster; some paradigms may operate on two-dimensional data, while others may use three-dimensional data.

[0024] Due to the different benefits and disadvantages of different paradigms, some paradigms are more suitable for different operating environments of the visual positioning system. For example, a visual positioning system may implement a paradigm that provides the necessary accuracy for situations where accuracy is required. Alternatively, a visual positioning system may implement a faster paradigm where considerations of operating speed outweigh the need for accuracy. In another example, some objects may operate at high altitudes, where an image capture device acquires images of the ground below. Due to the high altitude, the acquired images may appear as two-dimensional data frames. However, as the object moves closer to the ground, the environment may provide more and more three-dimensional data.

[0025] In certain embodiments, the visual localization system unifies different paradigms within a single visual localization framework. In particular, the unified visual localization framework can implement several paradigms within the navigation system that can determine the pose of an object based on a comparison of an acquired query image with retrieved image data. Thus, the navigation system can execute a visual localization paradigm that is appropriate for the specific operating environment of the object.

[0026] Figure 1 1 is a block diagram illustrating a potential navigation system 100 that implements a unified visual positioning architecture. For example, the navigation system 100 includes one or more image sensors 101 (also referred to herein as image acquisition devices), one or more processors 103, and a memory 105. In addition, the memory 105 can store an image database 107, a rough model 109, and paradigm information 111. In some alternative embodiments, the navigation system 100 may include additional sensors 113 and a user interface 115. As described herein, the user interface 115 can be any device capable of receiving user input and converting it into data that can be processed by the processor 103 and displaying information from the navigation system 100 to the user. Examples of the user interface 115 include a monitor, a touch screen, a smart phone, a microphone, a keyboard, a mouse, etc.

[0027] In some embodiments, the navigation system 100 can be securely mounted to an object. The object can be any object that can move or is being moved. For example, the object can be a vehicle, such as an aircraft, a train, a car, a ship, a spacecraft, etc. When the object is a vehicle, the vehicle can be manned or unmanned. In addition, the object can be a personal electronic device or other object that can be moved.

[0028] In an exemplary embodiment, the navigation system 100 is mounted to the object in a known positional relationship. The known positional relationship allows the navigation system 100 to derive the posture of the object from the detected position and posture of the sensors on the navigation system 100 (i.e., the image sensor 101 and the additional sensor 113). For example, the navigation system 100 is capable of determining the posture of the object in up to six degrees of freedom relative to a particular reference system. When the reference system is the earth reference system, the navigation system 100 can determine the latitude, longitude, and altitude of the object. In addition, where applicable to the object, the navigation system 100 can determine the posture of the object, such as yaw, pitch, and roll.

[0029] In further embodiments, the image sensor 101 may be a device that acquires image data from the environment surrounding the navigation system 100. For example, the image sensor 101 may be a camera, a depth camera, a laser radar, or other image acquisition device. The image sensor 101 may acquire image data as a query frame of image data. The image sensor 101 may be coupled to one or more processors 103, and the image sensor 101 provides the acquired image data to the one or more processors 103.

[0030] In an exemplary embodiment, the processor 103 receives the acquired image data (query frame) from the image sensor 101, and executes instructions that cause the processor 103 to execute a unified visual positioning architecture. The processor 103 can be a computing device (such as a general-purpose processor, a multi-core processor, a plurality of processors, a dedicated circuit, etc.). The functions performed by the processor 103 can be implemented using software, firmware, hardware, or any suitable combination thereof. The processor 103 and other computing devices can be supplemented or incorporated into a specially designed application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processor 103 and other computing devices may also include software programs, firmware, or other computer-readable instructions or run with them to perform various processing tasks, calculations, and control functions used in the present method and system.

[0031] The method can be implemented by computer executable instructions (such as program modules or components) executed by processor 103 or other computing devices. Typically, program modules include routines, programs, objects, data components, data structures, algorithms, etc. that perform specific tasks or implement specific abstract data types.

[0032] The memory can be any suitable computer-readable storage medium including, for example, nonvolatile memory devices, including semiconductor memory devices, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), or flash memory devices; disks, such as internal hard disks or removable disks; optical storage devices, such as compact disks (CDs), digital versatile disks (DVDs), Blu-ray disks, or other media that can carry or store desired program code as computer-executable instructions or data structures.

[0033] In certain embodiments, the processor 103 may be additionally coupled to a memory 105, wherein the memory 105 stores information used by the processor 103 to determine posture information from the acquired image data from the image sensor 101. For example, in addition to computer-readable instructions, the memory 105 also stores an image database 107. The image database 107 is a database of images received from a central repository of image data. The image database may include image data of the environment through which the navigation system 100 travels. For example, when the object is an aircraft, the image data may include overlapping, tiled satellite images of the earth's surface on which the aircraft can travel. In addition, the overlapping, tiled satellite images can be enhanced with three-dimensional information. In addition, the image database 107 may also include three-dimensional point cloud information of the environment in which the object can navigate.

[0034] In further embodiments, the image database 107 may include pre-computed coarse descriptors for each image in the image database 107. For example, for the coarse descriptors, the image database 107 may store a general descriptor for each image in the image database 107. Additionally, the image database 107 may store other descriptors that support the performance of various paradigms provided by the navigation system 100. For example, the image database 107 may include point cloud information, feature descriptors, and other descriptors that support the performance of various paradigms of the navigation system 100.

[0035] In further embodiments, the navigation system 100 obtains information in the image database 107 from a central repository, such as an operations center, a data center, or other location. Alternatively, the navigation system 100 may generate image information during normal operation and store the generated image information in the image database 107 for future use. When the navigation system 100 obtains information from another system, the navigation system 100 may receive the information to be stored in the image database 107 through a wireless or wired communication channel established between the navigation system 100 and the other system. Alternatively, when the navigation system 100 is remotely located from the central repository, the image data may be sent to another computer near the navigation system 100 by using a computer-readable medium or through a wireless or wired communication channel to be installed on the navigation system 100.

[0036] In certain embodiments, during operation, the image sensor 101 acquires a frame of image data (referred to as a "query frame") and provides the query frame to the processor 103. Upon receiving the query frame, the processor 103 prepares the query frame for comparison with the image data in the image database 107. In order to prepare the query frame for comparison, the processor 103 may execute the coarse model 109 to determine a series of coarse descriptors for the query frame. For example, the coarse model 109 may generate a general descriptor for the query frame. In some implementations, the general descriptor may be a vector of values ​​generated from the query frame. In some implementations, the coarse model 109 may be a machine learning model that calculates a general descriptor comparable to the coarse descriptors of the image data stored in the image database 107. In some implementations, the coarse model 109 may be the same model used to create the pre-computed coarse descriptors stored in the image database 107. Alternatively, the coarse model 109 may be a different model that produces a descriptor comparable to the coarse descriptors of the image data stored in the image database 107.

[0037] In addition, the memory 105 may also store paradigm information 111. The paradigm information 111 may enable the processor 103 to prepare the query frame for comparison with data in the image database within the selected visual localization paradigm. For example, the paradigm information 111 may use a feature recognition algorithm to identify features. In some specific implementations, the paradigm information 111 may be a fine model that receives the query frame as input and generates a fine descriptor of the input query frame. The paradigm information 111 may include a machine learning model that calculates descriptors comparable to descriptors of image data stored in the image database 107 for different visual localization paradigms. In some specific implementations, the same models may be used to generate paradigm-specific descriptors for query images that were used to generate any pre-computed descriptors of the associated paradigm for the image data stored in the image database 107. Alternatively, different models may generate paradigm-specific descriptors for query images from those models used to generate comparable descriptors for image data stored in the image database 107.

[0038] In some embodiments, when the navigation system 100 acquires image data, the navigation system 100 acquires the coarse model 109 and the paradigm information 111 from a central repository that provides the image data in an image database. Alternatively, when the coarse model 109 and the paradigm information 111 are substantially static, the coarse model 109 and the paradigm information 111 may be loaded onto the memory 105 prior to the initial operation of the navigation system 100. However, some paradigm information 111 may be acquired together with the image data or prior to the initial operation of the navigation system 100. When the coarse model 109 and the paradigm information 111 need to be updated or reinstalled in the memory 105, the navigation system 100 may acquire the coarse model 109 and the paradigm information 111 through a wireless or wired communication channel with the central repository. Alternatively, when the navigation system 100 is remotely located from a central repository, the coarse model 109 and paradigm information 111 may be transmitted to another computer proximate to the navigation system 100 using a computer readable medium or through a wireless or wired communication channel to be installed on the navigation system 100 .

[0039] In certain embodiments, when the processor 103 determines a coarse descriptor for the query frame by executing the coarse model 109 using the query frame as input, the processor 103 may compare the coarse descriptor with the coarse descriptors of the image data stored in the image database 107. The processor 103 then identifies a subset of images in the image database 107 that most closely match the query frame based on the comparison of these coarse descriptors. For example, the processor 103 may take a dot product of the coarse or global descriptor of the query frame with the global descriptor of each image or subset of images in the image database 107. Based on the results of the dot product, the processor 103 may identify a number of coarsely matched images in the image database 107 that best match the query frame. For example, the processor 103 may identify 20 (or other number) of the best matching images based on the dot product of the coarse descriptors. The number of coarsely matched images may be predefined or dynamically selected based on the situation of the navigation system 100.

[0040] In some embodiments, the number of roughly matching images in the image database 107 that are identified as the best match may be adjustable. For example, a user may communicate with the processor 103 via the user interface 115 to control the number of images selected. Alternatively, the number of images selected from the image database 107 may be preconfigured or controlled by another external system. By identifying a group of roughly matching images, non-matching images may be excluded from additional processing, thereby increasing processing speed because the group of roughly matching images is processed compared to processing all (or a large number) of images in the image database 107.

[0041] In some embodiments, when the processor 103 has selected a roughly matching image, the processor 103 may then execute one or more of a plurality of visual localization paradigms to identify the posture of an object associated with the navigation system 100. The processor 103 determines which paradigms to execute based on the operating environment of the object and the navigation system. For example, certain paradigms are more suitable for certain operating environments. For example, the operating environment may require faster posture calculations, more accurate posture calculations, operations in a two-dimensional environment, or operations in a three-dimensional environment. The processor 103 may identify a paradigm for a certain operating environment by receiving a paradigm selection from a user via the user interface 115. Alternatively, the processor 103 may receive a paradigm selection from another system that communicates with the navigation system 100. In addition, the processor 103 may obtain information from additional sensors 113, thereby allowing the processor 103 to determine various aspects of the operating environment.

[0042] In some embodiments, as described above, the navigation system 100 may include additional sensors 113. The additional sensors 113 include sensors that provide information that the processor 103 can use to determine position, attitude, heading, and altitude. For example, the additional sensors 113 may include an inertial measurement unit, a GPS receiver, a barometer, a magnetometer, an altimeter, etc. The additional sensors provide measurements of the movement of the object or environmental information, and the processor 103 uses this information to calculate the navigation information of the object. The processor 103 can use the navigation information to identify the operating environment for selecting a paradigm. In addition, the processor 103 can use the navigation information to improve the efficiency of executing the selected paradigm.

[0043] As described above, identifying a set of potential matching images from a database, selecting a paradigm for visual localization, and performing the selected visual localization using the set of potential matching images enables a unified visual localization architecture. In such an architecture, processor 103 can execute a visual localization paradigm suitable for the operating environment of navigation system 100 within a single system. Therefore, navigation system 100 can be used by a wider range of systems, and navigation system 100 can provide visual localization information in a wider range of environments and for a wider range of purposes.

[0044] Figure 2 2 is a block diagram of a system 200 for preparing an image database for use by a system such as the navigation system 100 that can provide visual positioning using multiple visual positioning paradigms within a unified visual positioning architecture. As shown, the system 200 includes an image source 221 and a server 223. The server 223 acquires images from the image source 221 and prepares the image information for storage in the image database 107 on the navigation system 100. To prepare the images for storage in the image database 107, the server 223 may include a memory 225, one or more processors 227, and a transceiver 229.

[0045] In some embodiments, image source 221 can be any source of image data captured on an area. For example, the image data can be a satellite map acquired from a third party, such as USGS NAIP or USGS HRO imagery. Alternatively, the image data can include a point cloud acquired using a lidar (or other three-dimensional vision acquisition device) for a specific area. In addition, the image data can include multiple images taken of a specific environment, which can also be used to create a three-dimensional point cloud or for other pose estimation algorithms.

[0046] In an exemplary embodiment, the server 223 processes the image data so that the image data is suitable for comparison with a query frame using a different paradigm. For example, the server 223 may receive USGS satellite image data covering approximately 12,000 square kilometers. The server 223 may process the image to create tiles with fifty percent overlap at five zoom levels. The server 223 may store both the image received from the image source 221 and the processed image tiles in an image database 231 on the memory 225. In an additional specific implementation, the server 223 may obtain multiple overlapping images of an environment that the processor 227 may process to create a point cloud for a particular environment. For example, the processor 227 may implement a motion structure recovery (SFM) algorithm to create a three-dimensional point cloud for the environment. The processor 227 may store the generated point cloud in the memory 225. In addition, the processor 227 may store the image and lidar data obtained from the image source 221 together with the location information associated with the obtained image data in the image database 231 on the memory 225.

[0047] In additional embodiments, when the processor 227 creates different tiles of image data, the processor 227 may calculate additional vertical information so that the tiles of image data contain three-dimensional data. For example, when the tiles of image data are geographic images, the processor 227 may identify vertical geographic information (terrain height) for the tiles of image data. The processor 227 may determine the terrain height of features in the tiles of image data using terrain height information provided by the USGS Shuttle Tomography (STRM) database or the USGS National 3D Elevation Program (3DEP) database. When the terrain height has been applied to the tiles of image data, the processor 227 may store the three-dimensional tiles in the image database 231.

[0048] The memory 225 may operate similarly to the memory 105 described above, except that the memory 225 stores more and different types of information. Since processing images acquired from the image source 221 may be a computationally intensive task, the server 223 may be a supercomputer having a processor 227 having multiple cores or multiple processors 227. The processor 227 may operate similarly to the processor 103 described above, except that the processor 227 may include more processors and process different data types.

[0049] In some embodiments, the processor 227 may also calculate descriptors for tiles of image data stored in the image database 231. Pre-calculating descriptors on the server 223 may reduce the computational load on the navigation system 100 and increase the speed at which the navigation system 100 may determine the pose of an object. To pre-calculate descriptors, the processor 227 may perform the above combined Figure 1 In some implementations, the rough model 109 and the paradigm information 111 stored in the memory 225 may be stored in the memory 225. Figure 1 The rough model 109 and the paradigm information 111 stored in the memory 105 are not the same.

[0050] As described above, the execution of the coarse model 109 generates a global descriptor of the image data. The processor 227 may store the generated global descriptor in the memory 225 as the global descriptor 233. In addition, the execution of the paradigm information 111 generates a paradigm-specific descriptor of the image data. The processor 227 may store the generated descriptor in the memory 225 as the paradigm descriptor 235.

[0051] When the descriptor is generated and the image data is prepared, the processor 227 can transmit the above Figure 1 The image data described in the above is transmitted to the navigation system 100 for storage in the image database 107. For example, the processor 227 may send all or a portion of a tile of the three-dimensional image data to the navigation system 100 or a point cloud associated with the expected environment of the navigation system 100. In addition, the processor 227 may send the global descriptor 233 and the paradigm descriptor 235 associated with the transmitted image data to the navigation system 100. In some specific implementations, the processor 227 may also send the coarse model 109 and the paradigm information 111 to the navigation system 100 for use in calculating the descriptor of the acquired query frame. Therefore, the server 223 can provide information that enables the navigation system 100 to perform multiple visual localization paradigms, such as the above combined Figure 1 described.

[0052] Figure 3is a flow chart of a method 300 for executing one of a plurality of visual localization paradigms within a unified visual localization framework. The steps of the method 300 are referred to herein above with respect to Figure 1 The method 300 continues at 301, where an image is obtained from a database. For example, the processor 103 may obtain images from the image database 107, where the images are associated with specific locations. In addition, the processor 103 may obtain general and canonical descriptors of image data stored in the image database 107. Furthermore, the method 300 continues at 303, where a query frame is obtained from an image sensor. For example, the processor 103 may obtain a query frame from the image sensor 101.

[0053] In some embodiments, when the processor 103 has obtained the query frame and the image from the image database 107. The method 300 continues at 305, where a set of roughly matching images is selected. For example, the processor 103 generates a generic descriptor for the query frame using the coarse model 109, and the processor 103 compares the query frame generic descriptor with the generic descriptors of the images in the image database 107. Based on the comparison, the processor 103 identifies a set of roughly matching images. When the processor 103 has identified the set of roughly matching images, the method 300 continues at 307, where a visual localization paradigm is selected based on the operating environment. For example, the processor 103 may receive an instruction from a user to select a visual localization paradigm, or the processor 103 may select a visual localization paradigm based on the detected operating environment. When the visual localization paradigm is selected, the method 300 continues at 309, where the posture of the object is calculated using the selected visual localization paradigm.

[0054] Figure 4 4 is a block diagram illustrating a multi-vision localization paradigm architecture 400. As shown, the architecture 400 may include a common front end 401 and a paradigm execution portion 403. The common front end 401 provides image information 413 as output that the paradigm execution portion 403 may use to recognize a pose. In addition, the common front end 401 may be unaware of the paradigm selected by the paradigm execution portion 403. The paradigm execution portion 403 uses the image information 413 provided from the common front end 401 to execute the selected vision localization paradigm to determine poses and other vision localization tasks.

[0055] In certain embodiments, the common front end 401 includes a candidate image identifier 405. The candidate image identifier 405 is a function executed by the processor 103 to reduce the number of images used by the selected paradigm so that the paradigm can quickly calculate the pose of the object. Without reducing the number of images to be used by the selected paradigm, the selected paradigm can perform visual localization on the body of image data stored in the image database, which may take an excessively long time. Therefore, the candidate image identifier 405 provides a faster mechanism for the multi-visual localization paradigm architecture 400 to reduce the number of images provided to the paradigm execution portion 403 as part of the image information 413 so that the paradigm execution portion 403 can quickly calculate the pose of the object.

[0056] The candidate image identifier 405 may receive several inputs to create the image information 413. For example, the candidate image identifier 405 may receive the query frame 407, the image data from the image database 409, and the model information 411. The candidate image identifier 405 may compare the query frame 407 with the image data in the image database 409 to identify the image subset. In addition, the candidate image identifier 405 may also use the model information 411 to convert the data in the image subset into a potential vision-based localization suitable for execution within the paradigm execution portion 403.

[0057] The candidate image identifier 405 receives the query frame 407 from the image sensor 101 to identify a subset of images that potentially match the query frame 407. The candidate image identifier 405 then computes a query coarse descriptor for the query frame 407 and compares the query coarse descriptor to a database coarse descriptor of a large set of image data stored in an image database 409. The candidate image identifier 405 selects a defined number of best matching images from the image database 409. In some implementations, after identifying the best matching images, the candidate image identifier 405 may apply model information 411 to the best matching images and the query frame 407 to generate representations of the query frame 407 and the best matching images that are suitable as inputs to the potential paradigm executed within the paradigm execution portion 403. For example, the model information 411 may cause the candidate image identifier 405 to generate local descriptors for the best matching images and the query frame 407, combine images from the best matching images, generate point clouds, perform feature recognition, etc. In an alternative implementation, the query frame 407 used as input to a particular formula and the generation of the representation of the best matching image may be performed by the associated formula within the formula execution portion 403 .

[0058] In some embodiments, after candidate image identifier 405 has received query frame 407 and selected the best matching image from image database 409, candidate image identifier 405 may provide this information to pattern execution portion 403 as image information 413. When candidate image identifier 405 also generates representations of query frame 407 and the best matching image suitable as input to a potential pattern executed within pattern execution portion 403, candidate image identifier 405 may also include the generated representations in image information 413 provided to pattern execution portion 403.

[0059] In certain embodiments, within the paradigm execution portion 403, the processor 103 may receive a selection from a user via the user interface 115. Alternatively, the processor 103 may identify a particular paradigm based on the operating environment of the object. In some implementations, the processor 103 may identify a particular paradigm based on a combination of user input and the detected operating environment. When the processor 103 identifies a paradigm to be executed, the processor 103 then uses the provided image information 413 to execute the identified paradigm. Furthermore, in some implementations, more than one paradigm may be selected for execution by the processor 103.

[0060] As shown, the paradigm execution part 403 includes multiple executable paradigms. In particular, the paradigm execution part 403 includes the first paradigm 415, the second paradigm 417, the third paradigm 419, the fourth paradigm 421, the fifth paradigm 423 and the sixth paradigm 425. Although six paradigms are shown, the paradigm execution part 403 can potentially provide more optional paradigms or less paradigms. Each paradigm can provide various benefits. For example, some paradigms can be executed more quickly, some paradigms can provide better accuracy, and some paradigms can be more suitable for performance in certain operating environments.

[0061] In certain embodiments, some paradigms may identify pose information of an object. In contrast, other paradigms may identify corresponding features between the query image and the image in the best matching image provided as part of the image information 413. For example, as shown, the third paradigm 419, the fourth paradigm 421, and the fifth paradigm 423 provide examples of paradigms that identify corresponding features. When a paradigm identifies corresponding features, the paradigm may provide information about the corresponding features to a pose solver 427. The pose solver 427 may use the information about the corresponding features and the information about the image sensor 101 to calculate the pose of an object associated with the navigation system 100.

[0062] In some embodiments, the query frame 407 may be a two-dimensional image, and the image in the best matching image from the image database 409 may contain three-dimensional data. Based on the correspondence between the two-dimensional image and the three-dimensional image based on the corresponding features, a perspective n-point (PnP) algorithm (such as EPnP, SQPnP, and P3P) may be used to determine the pose of the object associated with the two-dimensional image. In particular, when using the PnP algorithm, the processor 103 solves the following problems for matching features:

[0063]

[0064] in is the observed pixel position, is a known property of the image sensor 101 that defines the relationship of the image sensor 101 to an object attached to the navigation system 100. In addition, refers to the three-dimensional coordinates of the feature in the matching image associated with the observed pixel position in the earth coordinate system. Using this data, the processor 103 can estimate It represents the pose of an object attached to the navigation system 100 in six degrees of freedom. In particular, represents the roll, pitch, and yaw of the object, and Represents the position of the object in the reference frame used to match features in the image. Thus, processor 103 can calculate the pose of the object in six degrees of freedom. When processor 103 determines the pose of the object, processor 103, another processor on the object, or a user can use the pose to guide the navigation of an attached object or use the pose information for other purposes.

[0065] Figure 54 is a diagram illustrating an exemplary embodiment of a rough comparison of a query frame 407 obtained with a set of database images 501 in an image database 409 to identify a set of roughly matching images 509. For example, the candidate image identifier 405 may obtain the query frame 407 from the image sensor 101 and obtain a set of database images 501 from the image database 409. The candidate image identifier 405 may generate a general descriptor of the query frame 407 by providing the query frame 407 as an input to a rough model 503. The rough model 503 may operate similarly to the rough model 109 described above. Therefore, in some specific implementations, the rough model 503 may be a machine learning model that receives an image data frame as input and provides a column vector as an output. One such example of a machine learning model may be created using a residual neural network (ResNet), wherein a generalized average pooling (GeM) algorithm may be performed on the last layer of the ResNet, wherein the GeM is performed with the maximization of average precision as the goal. Other algorithms for generating global descriptors may also be used, such as a triplet loss method or other algorithms. For a model that can generate descriptors, the coarse model 503 can receive the query frame 407 as input and generate a query general descriptor 505 as output.

[0066] In certain embodiments, the candidate image identifier 405 may compare the query generic descriptor 505 with the generic descriptors 507 of the set of database images 501. In some implementations, the coarse model 503 may receive each image in the set of database images 501 as input individually and generate a generic descriptor for each image in the set of database images 501. In alternative implementations, the generic descriptors 507 for the set of database images 501 may be pre-computed and stored in the image database 409. Thus, the candidate image identifier 405 may pull the stored generic descriptors 507 for each image in the image database 409 from the image database 409.

[0067] In further embodiments, the query generic descriptor 505 and the generic descriptors 507 may be quickly compared to one another. For example, the same model used to generate the query generic descriptor 505 may be used to generate the generic descriptor 507. Alternatively, different models may be used to generate the query generic descriptor 505 and the generic descriptors 507, but the descriptors generated by the different models may still be comparable to one another. In one such implementation with comparable descriptors, the query generic descriptor 505 and the generic descriptors 507 for each image in the set of database images 501 may be represented as column vectors of the same dimension. Thus, the query generic descriptor 505 may be quickly compared to the generic descriptors 507 for a particular image in the set of database images 501 by taking the dot product of the query generic descriptor 505 and the generic descriptor 507 for the particular image. After the dot product has been calculated for the query universal descriptor 505 and the universal descriptor 507 of each image in the set of database images 501, the candidate image identifier 405 then identifies a defined number of images in the set of database images 501 having the highest resulting dot product as the set of roughly matching images 509. The defined number of images may be predefined, adjustable by a user, or depend on the operating environment of an object associated with the navigation system 100. As described above, for the set of roughly matching images 509, the candidate image identifier 405 may provide the set of roughly matching images 509 as part of the image information 413 to the paradigm execution portion 403. Using the image information 413, the paradigm execution portion 403 then provides the set of roughly matching images 509 as input to one or more of the selected visual localization paradigms.

[0068] Figure 6 419. As shown, the third paradigm 419 can use landmark navigation to identify the pose of the object. In particular, the third paradigm 419 can use landmark navigation to identify a candidate image from the identified set of roughly matching images 509 provided within the image information 413.

[0069] In some embodiments, the third paradigm 419 may identify a candidate image 607 from the group of roughly matched images 509 that match the query frame 407 in the image information 413 based on the local descriptors generated by the local descriptor model 601 included in the above-mentioned paradigm information 111. For example, the third paradigm 419 may obtain the query frame 407 and provide the query frame 407 as an input to the local descriptor model 601 to generate the query local descriptor 603. The local descriptor model 601 is an algorithm or other executable model that generates a local descriptor of an image based on the provided image data. In addition, the third paradigm 419 may provide each image in the group of roughly matched images 509 as an input to the local descriptor model 601 to identify the database local descriptor 605 of each image in the group of roughly matched images 509. The images in the group of roughly matched images 509 may be provided as input to the local descriptor model 601. At the same time, the local descriptor model 601 generates the query local descriptor 603 to generate the database local descriptor 605. Alternatively, database local descriptors 605 may be pre-computed by the system or another system and then stored in image database 409. When database local descriptors 605 are stored in image database 409, database local descriptors 605 may be provided for use by third normal form 419 as part of image information 413. In an alternative implementation, database local descriptors 605 may be calculated by candidate image identifier 405 and provided to third normal form 419 as part of image information 413.

[0070] In some embodiments, executing the local descriptor model 601 by the third paradigm 419 may include executing a learning-based local descriptor identification algorithm. For example, the learning-based local descriptor algorithm is a deep learning-based algorithm that extracts local descriptors (key points and descriptors) from an image. Therefore, each image in the set of roughly matched images 509 and the query frame 407 may be provided as input to a learning-based local descriptor algorithm executed by the processor or external system executing the third paradigm 419 as the local descriptor model 601. Therefore, the learning-based local descriptor algorithm may generate local descriptors for the query frame 407 and each image in the set of roughly matched images 509. Alternatively, the local descriptors of the query frame 407 may be generated using an algorithm different from the algorithm used to generate the local descriptors of the images in the set of roughly matched images 509, as long as the descriptors generated by the different algorithms are comparable to each other.

[0071] In some embodiments, the third normal form 419 compares the query local descriptor 603 to the database local descriptors 605 for each image in the set of roughly matched images 509 to identify the candidate image 607. For example, the system may provide the query local descriptor 603 and the database local descriptors 605 as inputs to an additional model that provides confidence measures for putative matches between images in the set of roughly matched images 509 and the query frame 407. The system may sum the resulting confidence measures to identify the candidate image 607 from the set of roughly matched images 509, where the candidate image 607 has the highest sum of confidence measures.

[0072] In certain embodiments, to determine which database local descriptors 605 best match the query local descriptors 603, the third paradigm 419 may perform an attention graph neural network plus a best match layer. For example, the third paradigm 419 may perform a graph neural network algorithm to compare the query local descriptor 603 with the database local descriptors 605 of each image in the set of roughly matched images 509. In addition, the graph neural network algorithm may be performed for different orientations and sizes of different roughly matched images to account for different orientations of objects relative to the environment and distances from surfaces in the environment. The graph neural network algorithm may refer to a deep learning-based algorithm that performs feature matching and geometric verification. The graph neural network algorithm obtains local descriptors from different images and matches the local descriptors using a graph neural network. The graph neural network algorithm generates a confidence measure for the assumed match. The third paradigm 419 may add the generated confidence measures to identify candidate images from the set of roughly matched images 509.

[0073] In some embodiments, when third paradigm 419 has identified candidate image 607, third paradigm 419 may provide information about corresponding features in candidate image 607 and query frame 407 to pose solver 427 along with information describing characteristics of image sensor 101. As described above, pose solver 427 may use PNP or other algorithms to calculate the pose of the object based on the correspondence between the identified features in query frame 407 and candidate image 607.

[0074] However, in some cases, the third paradigm 419 may identify an incorrect candidate image 607. Therefore, the third paradigm 419 may perform a validity check to determine whether the identified candidate image 607 is valid before providing the information to the pose solver 427. Fig. 7A and Figure 7B Validity checks are shown applied to candidate images 607. In particular, Fig. 7A shows the application of a homography matrix to a query frame 407 that maps the query frame onto a valid candidate image 607, and Figure 7BApplication of a homography matrix to a query frame 407 mapping the query frame onto an invalid candidate image is shown.

[0075] As stated, Fig. 7A Validity check is shown applied to candidate image 607. When performing the validity check, the system can identify a homography matrix based on the query local descriptors 603 and at least some of the database local descriptors 605. The system can use the identified homography matrix to create a mapping 701 of the boundaries of the query frame 407 onto the candidate image 607. After creating the mapping 701, the system can use the homography matrix to map the query local descriptors 603 onto the candidate image 607. Fig. 7A As shown, when the homography matrix is ​​applied to the query local descriptor 603, the query local descriptor 603 is mapped to the database local descriptor 605 within the mapping 701. Because the mapped database local descriptor 605 is located within the mapping 701, Fig. 7A The candidate image 607 in passes the validity check, and the candidate image 607 is designated as a matching image.

[0076] Instead, in creating a mapping similar to 701 from the homography matrix Figure 7B In the mapping 701, some query local descriptors 603 are mapped to database local descriptors 605 outside the mapping 701. When the query local descriptor 603 is mapped to the database local descriptor 605 outside the mapping 701, Figure 7B 701 does not pass the validity check. In some implementations, when any query local descriptor 603 maps to a database local descriptor 605 outside of the mapping 701, the candidate image 607 may not pass the validity check. However, when a certain number or proportion of query local descriptors 603 map to database local descriptors 605 outside of the mapping 701, the candidate image 607 may not pass the validity check.

[0077] In addition, when the mapping 701 created from the homography matrix represents an impossible potential mapping 701 onto the query frame 407 onto the candidate image 607, the candidate image 607 may not pass the validity check. For example, the mapping 701 may be an extreme non-rectangular mapping or a messy mapping. When the mapping 701 is an impossible mapping, the messy mapping or irregular shape may indicate that the homography matrix obtained from the local descriptor is invalid. Therefore, the candidate image 607 is a wrong match for the query frame 407. When the third paradigm 419 determines that the candidate image 607 is a wrong match, the third paradigm 419 may select the next best matching image that can pass the validity check in the set of roughly matched images 509.

[0078] Figure 8is a diagram illustrating a visual localization paradigm such as the fourth visual localization paradigm 421. As shown, the fourth paradigm 421 can use a structure from motion (SfM) algorithm to recognize the pose of an object. In the SfM paradigm, a processor can calculate frames of three-dimensional information 803 based on a sequence of two-dimensional images acquired from a mobile image capture device (such as a camera moving through an environment). The three-dimensional information can then be saved on an image database 409. The three-dimensional information can then be provided to a rough model 503 to calculate a common descriptor for a set of information. The image database 409 can contain multiple frames of three-dimensional information, and the set of roughly matched images 509 can be selected from these three-dimensional information frames.

[0079] In some embodiments, the three-dimensional information frame may be selected from the set of roughly matching images 509. Similar to the third paradigm 419 described above, the candidate image may be selected based on comparing the local descriptors of the query frame 407 with the identified local descriptors of the three-dimensional information 803.

[0080] Within the SFM paradigm, the computational intensity of creating three-dimensional information increases as the area represented within the frame of three-dimensional information 803 becomes larger. For example, representing a home or family group within the frame of three-dimensional information 803 may be a feasible task for a computing device. However, representing a city or larger area within the frame of three-dimensional information 803 becomes computationally inaccessible to many supercomputers. Therefore, the SFM paradigm may be more suitable for smaller, primarily three-dimensional environments.

[0081] In an exemplary embodiment, a vehicle 801 having the navigation system 100 mounted thereon may move through an environment and capture a query frame 407. As described above, the navigation system 100 may identify a set of roughly matching images 509 in the image database 409 that potentially correspond to the query frame 407. Using the identified set of roughly matching images 509, the navigation system 100 may identify local descriptors in the set of roughly matching images 509 and local descriptors in the set of roughly matching images 509. The local descriptors may describe features represented in the query frame 407 and the set of roughly matching images 509. The local descriptors provide information about features within the set of roughly matching images 509. The navigation system 100 may then identify a frame of three-dimensional information 803 that most closely matches the query frame 407.

[0082] In some embodiments, using the query frame 407 and corresponding features in the three-dimensional information 803, the navigation system 100 can identify a mapping 805. The mapping 805 can be a mapping of the boundaries of the query frame 407 to the three-dimensional information 803. Alternatively, the mapping 805 can correspond to a mapping of features of the query frame 407 to corresponding features in the frame of the three-dimensional information 803.

[0083] In an exemplary embodiment, the mapping 805 may depend on the view angle 807 of the image sensor 101 of the navigation system 100 mounted to the vehicle 801. By knowing the view angle 807, the navigation system 100 can determine the pose of the vehicle 801 relative to the features identified in the frame of the three-dimensional information 803 with a known position. However, the view angle 807 depends on the characteristics of the image sensor 101 that captured the query frame 407, which are known a priori. Therefore, the navigation system 100 can calculate the view angle 807 of the image sensor 101 using information about the characteristics of the image sensor 101 and the corresponding features identified in the query frame 407 and in the frame of the three-dimensional information 803. Therefore, the information about the corresponding features and the information about the characteristics of the image sensor 101 can be provided to the pose solver 427, which uses this information to calculate the pose, as described above. Therefore, the fourth paradigm 421 can perform visual positioning using image retrieval from the image database 409.

[0084] Fig. 9 is a diagram illustrating additional visual localization paradigms such as the sixth visual localization paradigm 425. As shown, the sixth paradigm 425 can use neural rendering to identify the pose of an object. Neural rendering is an image and video generation technique that uses machine learning to generate new images based on acquired images. In the context of visual localization, neural rendering can implement a machine learning algorithm 903 to generate two-dimensional and three-dimensional representations 905 of a scene based on a set of previously acquired images 901. The generated representations 905 can be stored in an image database 409 for retrieval by the navigation system 100 when performing visual localization. In particular, the navigation system 100 can obtain a query frame 407 from the image sensor 101 and identify the pose of the image sensor 101 within the environment.

[0085] Using neural rendering can provide some benefits. For example, neural rendering can capture three-dimensional representations of scenes that are difficult or impossible to capture with traditional image capture methods. There are several advantages to using neural rendering for visual positioning. First, it can create three-dimensional representations of scenes that are difficult or impossible to capture using traditional image capture methods. For example, neural rendering can create three-dimensional representations of indoor or constantly changing scenes (such as busy streets). In addition, because the neural network machine learning algorithm 903 can learn to represent the appearance of an object at a fine level, neural rendering can be used to create more detailed three-dimensional representations. Fine and detailed three-dimensional representations can be used for visual positioning, which can help improve the accuracy of camera pose estimation.

[0086] However, like other paradigms, neural rendering also potentially has disadvantages. In particular, the quality of the model produced by algorithm 903 depends on the quality of the training data. A poor quality training set may create a poor quality generated representation. In addition, algorithm 903 may be computationally intensive. Therefore, the execution of algorithm 903 may require a supercomputer or other large computer to generate the desired representation.

[0087] For neural rendering, in the above Figure 2 The paradigm information 111 implemented on the system 200 described in the embodiment of the present invention may include instructions for directing the processor 227 to implement the algorithm 903 on a set of previously acquired images 901 from the image source 221. The algorithm 903 may be trained on quality training data so that the algorithm 903 can generate similar quality representations. For the generated model algorithm 903, the set of previously acquired images 901 may be provided as input to the algorithm 903. The set of previously acquired images 901 may be a plurality of two-dimensional or three-dimensional data acquired from a particular scene. Based on the input, the algorithm 903 generates a set of two-dimensional and three-dimensional representations 905 that the navigation system 100 can use to compare with the potential query frame 407. The set of two-dimensional and three-dimensional representations 905 may be loaded into the image database 409 on the navigation system 100.

[0088] In certain embodiments, when the image sensor 101 on the navigation system 100 acquires the query frame 407, the navigation system 100 can identify the set of roughly matching images 509. With the identified set of roughly matching images 509, the query frame 407 and the two-dimensional and three-dimensional representations 905 can be provided as input to the algorithm 903, which can determine the pose of the camera based on the modeled relationship of the query frame 407 and the two-dimensional and three-dimensional representations 905. Additionally, characteristics of the image sensor 101 can be provided as input to the algorithm 903.

[0089] Fig.10 423. As shown, the fifth paradigm 423 can use learned deep visual localization to identify the pose of an object. Learned deep visual localization is a technique that uses deep learning to estimate the camera pose of a query frame 407 by learning the relationship between depth and image features. Learned deep visual localization can constrain the possible poses of the image sensor 101 that may have captured an image of the object. For example, if the image represents an object that is known to be one meter tall, then the image constrains the image sensor 101 to the location where the image of the object may have been captured.

[0090] As stated, deep learning can be used to learn the relationship between depth and image features. To train the model, a deep neural network can be used on a dataset of images 1003 and their corresponding depth maps 1001. When training the model, the neural network learns to predict the depth of objects in the image based on the image features. Once the neural network is fully trained, the neural network can be used to estimate the camera pose of the query frame 407.

[0091] Like the other paradigms described in this article, the learning deep visual localization paradigm provides some advantages and disadvantages. For example, because learning deep visual localization uses depth information, it can be well suited for challenging environments that are subject to changes in lighting conditions and clutter. In particular, depth information can compensate for changes in scene appearance and help disambiguate different objects within the scene. If the resulting model is trained on data from a variety of environments, the model may be able to generalize to new environments, making learning deep visual localization well suited for localization in new environments. However, learning deep visual localization relies on real and diverse training data, and training the model and analyzing the acquired data frames can be computationally expensive. In addition, when there is noise in the depth information, the model generated using learning deep visual localization may be subject to errors.

[0092] In certain embodiments, the learned deep algorithm can estimate the pose of the image sensor 101 by identifying features corresponding to images with known depths retrieved from the image database 409. In particular, the set of roughly matched images 509 may have been previously processed using a deep learning neural network to estimate depth information of features in the set of roughly matched images 509. Using the set of roughly matched images 509, the navigation system 100 can compare the query frame 407 with images in the set of roughly matched images 509 to identify a candidate image 1005 in the set of roughly matched images 509. For example, the navigation system 100 can compare local descriptors of features in the query frame 407 with local descriptors of features of each image in the set of roughly matched images 509. Based on the comparison, the navigation system 100 can identify the candidate image 1005 as the image in the set of roughly matched images 509 having the highest matching metric, where the matching metric can be the sum of matching scores between features in the query frame 407 and images in the set of roughly matched images 509.

[0093] In certain embodiments, using the corresponding features 1007 in the query frame 407 and the candidate images 1005, the navigation system 100 can identify a mapping 1009. The mapping 1009 can be a map of the boundaries of the query frame 407 to the candidate images 1005. Furthermore, the mapping 1009 can be derived from the mapping of the corresponding features 1007 between the query frame 407 and the candidate images 1005.

[0094] In an exemplary embodiment, mapping 1009 may depend on the pose of the image sensor 101 of the navigation system 100 mounted to the object in the environment. By knowing the corresponding features 1007, the navigation system 100 can use the learned depth information of the corresponding features in the candidate image 1005 and their appearance in the query frame 407 to determine the pose of the object within the environment. Therefore, the navigation system 100 can use the information about the characteristics of the image sensor 101 and the identified corresponding features 1007 in the query frame 407 and in the candidate image to calculate the pose of the image sensor 101. Therefore, the information about the corresponding features 1007 and the information about the characteristics of the image sensor 101 can be provided to the pose solver 427, which uses this information to calculate the pose, as described above. Therefore, the fifth paradigm 423 may also be able to perform visual positioning using image retrieval from the image database 409.

[0095] Fig.11 407 , is a diagram illustrating an additional visual localization paradigm such as the first visual localization paradigm 415. As shown, the first paradigm 415 can use pose approximation visual localization to identify the pose of an object. Pose approximation visual localization is a technique for estimating the pose of an image sensor 101 by finding an image or combination of images that has a known pose that most closely matches a query frame 407 produced by the image sensor 101. Pose approximation relies on the assumption that each scene is unique. Therefore, if the scene represented in an image in the image database 409 matches the scene represented in the query frame 407 captured by the image sensor 101, then the pose of the image sensor 101 must match the pose of the image capture device that acquired the matching image in the image database 409.

[0096] In an exemplary embodiment, the image sensor 101 on the navigation system 100 may acquire a query frame 407 representing a scene of the environment 1101. As described above, the navigation system 100 includes an image database 409 storing a plurality of captured images representing different scenes within the environment 1101, wherein the poses of the image capture devices 1105-1–1105-3 that captured the scenes represented in the image database 409 are known. For example, the image database 409 may include images of different scenes acquired by different image capture devices 1105-1–1105-3, wherein the poses of the different image capture devices 1105-1–1105-3 are known when the different image capture devices 1105-1–1105-3 captured the images. When the navigation system 100 receives the query image and the set of roughly matching images 509, the navigation system 100 may identify the image in the set of roughly matching images 509 that is most similar to the query frame 407 captured by the image sensor 101. Navigation system 100 can then determine that the pose of image sensor 101 is the pose of image capture device 1105 that captured the closest image in the set of roughly matching images 509. For example, navigation system 100 can determine that the image in the set of roughly matching images 509 created by image capture device 1105-2 is closest to query frame 407. Therefore, navigation system 100 will determine that the pose of image sensor 101 is the same as the pose of image capture device 1105-2.

[0097] In some implementations, the navigation system 100 can determine a pose based on a combination of poses of different image capture devices 1105-1–1105-3 that captured scenes represented in images in the set of roughly matching images 509. For example, the navigation system 100 can calculate a weight for each image in the set of roughly matching images 509, where the weight represents the similarity of the query frame 407 to a particular image in the set of roughly matching images 509. The pose of the image sensor 101 can then be calculated for a particular dimension by summing the weights applied to the poses of the associated image capture devices 1105-1–1105-3. For example, the pose of the image sensor 101 can be calculated using the following equation:

[0098]

[0099] As shown, for each sensor i associated with an image retrieved from the image database 409, the pose along a particular dimension may be calculated as the sum of the weight w multiplied by the pose T.

[0100] Like the other paradigms described in this article, the pose approximation visual localization paradigm provides some advantages and disadvantages. For example, pose approximation is relatively simple and computationally efficient, making it suitable for situations with limited processing resources or fast pose calculations. In addition, it is less sensitive to occlusion and provides scalability to different levels of accuracy. However, pose approximation has some disadvantages because it does not take into account the full three-dimensional structure of the scene. For example, pose approximation may not be as accurate as other visual localization paradigms described in this article. In addition, pose approximation is more sensitive to changes in the environment.

[0101] Fig.12 417. As shown, the second paradigm 417 can use dual-view geometric visual localization to identify the pose of an object. Dual-view geometric visual localization is a technique that uses the geometric relationship between two images of the same scene to estimate the pose of the image sensor 101. For example, the navigation system 100 can use epipolar geometry to create a homography matrix that describes the transformation of the query frame 407 to the candidate image 1201 in the set of roughly matched images 509.

[0102] As described herein, homography describes the change in the perceived position of an object when the viewpoint of the image acquisition device changes. In particular, the change in viewpoint distorts the perceived position of the object. Homography techniques identify common points between different image projections, such as the image projections in the query frame 407 and the candidate image 1201. Kernel surface geometry allows a homography to relate any two images of the same planar surface, which allows the estimation of a homography matrix. Using this knowledge of the homography matrix, the navigation system 100 can identify a mapping 1209 of the query frame 407 onto the candidate image 1201. Using the mapping 1209, the homography matrix, and knowledge of the pose of the sensor that originally captured the candidate image 1201, the navigation system 100 can calculate the position of the image sensor.

[0103] To further illustrate, a first image 1203 of a scene is captured by an image sensor at a first position 1211, where the first position 1211 is known. A second image sensor may capture a second image 1205 of the scene at an unknown second position 1213. A processor performing dual view geometry visual positioning may identify corresponding points and surfaces in the first image 1203 and the second image 1205. Using this information, the processor may identify a transformation 1207 that describes the translation and rotation of the image sensor moving from the first position 1211 to the second position 1213, or vice versa. Thus, using the transformation 1207 and knowledge of the first position 1211, the processor may estimate the pose of the second image sensor at the second position 1213.

[0104] In certain embodiments, the navigation system 100 may identify a candidate image 1201 from the set of roughly matched images 509. After identifying the candidate image 1201, the navigation system 100 may use a dual-view geometric visual positioning paradigm to identify a transformation of the query frame 407 to a position representing the position of the image acquisition device that acquired the candidate image 1201. The navigation system 100 then uses the identified transformation and the position associated with the candidate image 1201 to calculate the pose of the image sensor 101 that acquired the query frame 407. In addition, the navigation system 100 may use the lens characteristics of the image sensor 101 to identify the pose of the attached object. Dual-view geometric visual positioning provides several advantages, including its simple implementation and accuracy within a constrained environment. However, dual-view geometric visual positioning is sensitive to noise and relies on observing planar surfaces within the corresponding scene.

[0105] Although the following visual positioning paradigm is described above, the navigation system 100 can also select from other available visual positioning paradigms. Examples of visual positioning paradigms include deep feature matching, posture graph optimization, geographic location-based visual positioning, visual simultaneous positioning and mapping (VSLAM), visual inertial odometer (VIO), active visual positioning, and feature-based visual positioning. In addition, developing vision and processing technologies can lead to new visual positioning paradigms. New visual positioning paradigms can be easily incorporated into the embodiments described herein. In particular, the navigation system 100 can select a new paradigm based on the top selected image. In addition, as processing power increases, the number of selected images provided to the selected paradigm can also increase.

[0106] Fig.13 1 is a flow chart of a method 1300 for implementing a unified visual localization architecture. The method 1300 continues at 1301, where a query frame is acquired from an image sensor mounted to an object. The method 1300 further continues at 1303, where image data is acquired from an image database. Additionally, the method 1300 continues at 1305, where image information is identified based on the image data and the query frame. Additionally, the method 1307 continues at 1300, where a visual localization paradigm is selected from a plurality of visual localization algorithms. Additionally, the method continues at 1309, where a visual localization paradigm is performed using the image information as input to identify a relationship between the image information and the query frame. Additionally, the method 1300 continues at 1311, where a pose of the object is calculated based on the relationship.

[0107] Example Implementation

[0108] Embodiment 1 includes a system comprising: an image acquisition device mounted to an object, the image acquisition device being configured to acquire a query frame of an environment containing the object; a memory device being configured to store an image database; and at least one processor being configured to execute computer-readable instructions that direct the at least one processor to: identify a set of data in the image database that potentially matches the query frame; identify a visual localization paradigm among a plurality of visual localization paradigms; and determine a posture of the object using the set of data, the query frame, and lens characteristics of the image acquisition device as input to the visual localization paradigm.

[0109] Embodiment 2 includes a system according to embodiment 1, wherein the computer-readable instructions that instruct the at least one processor to identify the group of data further instruct the at least one processor to: calculate a query universal descriptor for the query frame; obtain universal descriptors of multiple frames stored in the image database; for each frame in the multiple frames, compare the universal descriptor with the query universal descriptor; and designate a certain number of frames in the multiple frames as the group of data.

[0110] Embodiment 3 includes the system of embodiment 2, wherein the computer readable instructions directing the at least one processor to identify the set of data further directs the at least one processor to provide additional paradigm specific information for the certain number of frames included in the set of data.

[0111] Embodiment 4 includes a system according to any of Embodiments 1-3, further comprising a user interface, wherein the computer-readable instructions that direct the at least one processor to identify the visual localization paradigm further direct the at least one processor to receive a paradigm selection from the user interface.

[0112] Embodiment 5 includes a system according to any one of embodiments 1-4, further comprising an additional sensor, wherein the computer-readable instructions that instruct the at least one processor to identify the visual positioning paradigm further instruct the at least one processor to detect the operating environment of the object based on navigation information obtained from the additional sensor.

[0113] Embodiment 6 includes the system of embodiment 5, wherein the operating environment includes at least one of: desired accuracy; whether the environment is a two-dimensional environment or a three-dimensional environment; and available processing power.

[0114] Embodiment 7 includes the system of any one of embodiments 1-6, wherein the data stored on the image database is received from a central repository, wherein the data stored on the image database is calculated by a plurality of processors at the central repository.

[0115] Embodiment 8 includes a system according to any one of embodiments 1-7, wherein the visual localization paradigm is at least one of the following: a pose approximation visual localization paradigm; a dual-view geometry visual localization paradigm; a landmark navigation visual localization paradigm; a motion structure recovery visual localization paradigm; a learned deep visual localization paradigm; and a neural rendering visual localization paradigm.

[0116] Embodiment 9 includes a method, comprising: acquiring a query frame from an image sensor mounted to an object; acquiring image data from an image database; identifying image information based on the image data and the query frame; selecting a visual localization paradigm from a plurality of visual localization algorithms; executing the visual localization paradigm using the image information as input to identify a relationship between the image information and the query frame; and calculating a posture of the object based on the relationship.

[0117] Embodiment 10 includes the method according to embodiment 9, wherein obtaining the image data from the image database further includes: calculating the query universal descriptor of the query frame; obtaining the database universal descriptors of multiple frames stored in the image database; for each frame in the multiple frames, comparing the database universal descriptor with the query universal descriptor; and designating a certain number of frames in the multiple frames as the image data.

[0118] Embodiment 11 includes the method according to embodiment 10, wherein identifying the image information further comprises identifying additional paradigm specific information for the certain number of frames included in the image data.

[0119] Embodiment 12 includes the method according to any of embodiments 9-11, wherein selecting the visual positioning paradigm further includes receiving a paradigm selection from a user interface.

[0120] Embodiment 13 includes the method according to any one of embodiments 9-12, wherein selecting the visual localization paradigm further includes identifying the operating environment of the object based on information obtained from at least one sensor.

[0121] Embodiment 14 includes a method according to embodiment 13, wherein the operating environment includes at least one of the following: desired accuracy; whether the environment of the object is a two-dimensional environment or a three-dimensional environment; and available processing power.

[0122] Embodiment 15 includes the method according to any one of embodiments 9-14, further comprising receiving data stored on the image database from a central repository, wherein the data stored on the image database is calculated by multiple processors at the central repository.

[0123] Embodiment 16 includes a method according to any one of embodiments 9-15, wherein the visual localization paradigm is at least one of the following: a pose approximation visual localization paradigm; a dual-view geometry visual localization paradigm; a landmark navigation visual localization paradigm; a motion structure recovery visual localization paradigm; a learned deep visual localization paradigm; and a neural rendering visual localization paradigm.

[0124] Embodiment 17 includes a system comprising: an image sensor configured to acquire a query frame representing a scene in an environment of an object on which the image sensor is mounted; a memory device configured to store an image database; and at least one processor configured to execute computer-readable instructions that direct the at least one processor to: execute a common front end configured to receive the query frame from the image sensor; identify image information including a set of data acquired from the image database and the query frame; and provide the image information as output; and execute a paradigm execution portion configured to receive the image information from the common front end; identify a visual localization paradigm among multiple possible visual localization paradigms; execute the visual localization paradigm to identify a relationship between the query frame and the set of data; and calculate the posture of the object based on the relationship.

[0125] Embodiment 18 includes a system according to embodiment 17, wherein the data stored on the image database is received from a central repository, wherein the central repository includes: a plurality of processors; a repository image database, the repository image database storing a repository of image data acquired from a third party; and a transceiver for providing the data stored on the image database; wherein the plurality of processors use a portion of the repository of image data to execute a plurality of algorithms to create information that supports visual positioning in the plurality of possible visual positioning paradigms.

[0126] Embodiment 19 includes a system according to any one of embodiments 17-18, further comprising a user interface, wherein the paradigm execution portion is configured to identify the visual positioning paradigm by receiving a paradigm selection via the user interface.

[0127] Embodiment 20 includes a system according to any one of embodiments 17-19, wherein the visual localization paradigm is at least one of: a pose approximation visual localization paradigm; a dual-view geometry visual localization paradigm; a landmark navigation visual localization paradigm; a motion structure recovery visual localization paradigm; a learned deep visual localization paradigm; and a neural rendering visual localization paradigm.

[0128] Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. Therefore, it is manifestly intended that the present invention be limited only by the claims and the equivalents thereof.

Claims

1. A system, comprising: an image acquisition device mounted to the object, the image acquisition device being configured to acquire a query frame of an environment containing the object; a memory device configured to store an image database; and at least one processor configured to execute computer-readable instructions directing the at least one processor to: identifying a set of data in the image database that potentially matches the query frame; Identify the visual localization paradigm among multiple visual localization paradigms; as well as The pose of the object is determined using the set of data, the query frame, and lens characteristics of the image acquisition device as input to the visual localization paradigm.

2. The system of claim 1, wherein the visual localization paradigm is at least one of the following: The pose-approximation visual localization paradigm; Dual-view geometric visual localization paradigm; Landmark navigation visual localization paradigm; Structure from motion restoration visual localization paradigm; Learning a deep visual localization paradigm; and A paradigm for visual localization in neural rendering.

3. A method comprising: acquiring a query frame from an image sensor mounted to the object; Acquire image data from an image database; identifying image information based on the image data and the query frame; Select a visual localization paradigm from multiple visual localization algorithms; performing the visual localization paradigm using the image information as input to identify a relationship between the image information and the query frame; as well as A pose of the object is calculated based on the relationship.

Images