Computer-implemented method, computer system, and non-transitory computer-readable memory for feature matching using features extracted from perspective corrected image

Abstract

This invention discloses an image localization system that receives an image of a scene and generates a depth map of the image by inputting the image into a model trained to generate a depth map of the image. The system determines the surface normal vectors of the pixels in the depth map. The system clusters these surface normal vectors to identify regions in the image corresponding to a plane. The system segments the image into patches, each patch being a region of connected pixels in the image and corresponding to a cluster of surface normal vectors. The system corrects perspective distortion in the patches and extracts perspective correction features from the corrected patches. The system matches these perspective correction features of the image with perspective correction features of other images for 3D relocalization.

Description

[Technical Field]

[0001] The subject matter described is generally about computer vision, and more specifically, the subject matter described is about feature matching using features extracted from perspective-corrected images. [Previous Technology]

[0002] Problem

[0003] Feature matching is useful for 3D relocalization and multi-view reconstruction in various applications (such as augmented / virtual / mixed reality (e.g., parallel reality games), robot navigation (e.g., autonomous vehicles), etc.). However, the viewpoint and distance of an image can significantly affect the recognition rate of features within the image. Given the prerequisites for using such methods (such as collecting training data with large appearance variations), familiar methods for improving the appearance of images (such as by selecting better local feature points or utilizing external information) may be impractical. Improved feature matching techniques are needed to achieve 3D relocalization and multi-view reconstruction. [Summary of the Invention]

[0004] This invention describes a feature matching system that uses depth estimation to remove perspective distortion in an image. After removing the perspective distortion, the feature matching system extracts features from the image and uses these perspective correction features for feature matching.

[0005] In some embodiments, the feature matching system receives an image depicting a scene. The image may be a monocular image. The feature matching system generates a depth map of the image. For example, the feature matching module inputs the image into a neural network, and the neural network outputs the depth map. The depth map contains pixels, and the feature matching system estimates a surface normal vector for each pixel (or a subset of such pixels) in the depth map. The feature matching system clusters the pixels based on the surface normal vectors of such pixels. Since each pixel in the depth map corresponds to an element (e.g., a pixel) of the image, the feature matching system can cluster the pixels of the image based on the clusters of such pixels in the depth map. The feature matching system uses such clusters to segment the image into patches. Each patch corresponds to a cluster and contains a region of connected pixels in the image.

[0006] The feature matching system further corrects the perspective view of one of the patches and produces a corrected patch. In some embodiments, the feature matching system uses homography to perform a projection transformation. When applied to a patch, homography removes perspective distortion from the patch and transforms the patch into a corrected patch. The feature matching system extracts perspective correction features from the corrected patches.

[0007] This feature matching system matches the perspective correction features of the image with the perspective correction features of other images (or with a 3D model of a scene) to reposition an object in the scene or reconstruct the scene using a multi-view approach. Because perspective distortion is removed from the image by correcting the patches before feature extraction, the feature matching system can perform effective feature matching even under large perspective variations. Compared to conventional feature matching methods, this feature matching system achieves more robust feature matching, and therefore achieves better 3D repositioning and multi-view 3D reconstruction. [Simplified Explanation of the Diagram]

[0088] Figure 1 is a block diagram of a network computing environment suitable for hosting a parallel reality game according to an embodiment.

[0089] Figure 2 is a block diagram of a user terminal device of Figure 1 according to an embodiment.

[0090] Figure 3 is a block diagram of a game server of Figure 1 according to an embodiment.

[0091] Figure 4 is a block diagram of one embodiment of the feature matching system of Figure 1.

[0092] Figures 5A to 5E illustrate a pipeline for extracting perspective correction features from an image by the feature matching system in Figure 4 according to an embodiment.

[0093] Figure 6 depicts a representation of a virtual world with a geography parallel to the real world according to an embodiment.

[0094] Figure 7 is a flowchart illustrating one method for extracting perspective correction features from an image according to an embodiment.

[0095] Figure 8 is a high-level block diagram illustrating a functional view of a typical computer system used as a feature matching system of Figure 4 according to an embodiment.

Implementation Method

[0008] Cross-reference to related applications

[0009] This application claims the rights of U.S. Provisional Patent Application No. 63 / 064,621, filed August 12, 2020, the entire contents of which are incorporated herein by reference.

[0010] The figures and the following description illustrate certain embodiments only by illustrative means. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structure and methods may be employed without departing from the principles described. Where feasible, similar or identical element symbols are used in the figures to indicate similar or identical functionality. Where elements share a common numeral followed by a different letter, this indicates that the elements are similar or identical. Unless the context otherwise indicates, reference to any one of the numerals generally refers to any one or any combination of such elements.

[0011] Various embodiments are described in the context of a parallel reality game that includes augmented reality content in a virtual world geography parallel to at least a portion of the real world geography, such that player movement and actions in the real world affect actions in the virtual world and vice versa. Those skilled in the art using the disclosure provided herein will understand that the described subject matter can be applied to other situations where depth information is desired to be determined from image data. Furthermore, the inherent flexibility of computer systems allows for a variety of possible configurations, combinations, and divisions of tasks and functions among the components of the system. For example, systems and methods according to the present invention can be implemented using a single computing device or across multiple computing devices (e.g., connected in a computer network).

[0012] Figure 1 is a block diagram of a network computing environment 100 suitable for hosting a parallel reality game according to an embodiment. The network computing environment 100 provides interaction between players in a virtual world with a geographic location parallel to the real world. Specifically, a geographic area in the real world can be directly linked to or mapped to a corresponding area in the virtual world. A player can move around in the virtual world by moving to various geographic locations in the real world. For example, the location of a player in the real world can be tracked and used to update the location of the player in the virtual world. Typically, the location of a player in the real world is determined by finding the location of a user terminal device 110 through which the player interacts with the virtual world and assuming that the player is in the same (or approximately the same) location. For example, in various embodiments, if the location of a player in the real world is within a certain distance (e.g., ten meters, twenty meters, etc.) of the real-world location corresponding to the virtual location of a virtual element in the virtual world, then the player can interact with a virtual element. For convenience, various embodiments are described with reference to "player's location," but those skilled in the art will understand that such reference may refer to the location of the player's client device 110.

[0013] The network computing environment 100 uses a client-server architecture, wherein a game server 120 communicates with a client device 110 via a network 150 to provide a parallel reality game to the player at the client device 110. The network computing environment 100 may also include other external systems, such as sponsor / advertiser systems or commercial systems. Although only one client device 110 is shown in Figure 1, any number of client devices 110 or other external systems can be connected to the game server 120 via the network 150.

[0014] A client device 110 may be any portable computing device that can be used by a player to interface with the game server 120. For example, a client device 110 may be a wireless device, a digital assistant (PDA), a portable gaming device, a cellular phone, a smartphone, a tablet computer, a navigation system, a handheld GPS system, a wearable computing device, a display with one or more processors, or other such devices. In another example, the client device 110 includes a conventional computer system, such as a desktop computer or a laptop computer. However, the client device 110 may be a vehicle with a computing device. In short, a client device 110 may be any computer device or system that enables a player to interact with the game server 120. As a computing device, the client device 110 may include one or more processors and one or more computer-readable storage media. The computer-readable storage media may store instructions that cause the processors to perform operations. User device 110 is preferably a portable computing device, such as a smartphone or tablet, that can be easily carried or otherwise transported with a player. Various embodiments of user device 110 are described in more detail below with reference to FIG2.

[0015] The game server 120 may be any computing device and may include one or more processors and one or more computer-readable storage media. The computer-readable storage media may store instructions that cause the processors to perform operations. In various embodiments, the game server 120 maintains a replica of a main game state and manages the interaction between the player client devices 110 so that all players interact with a consistent game world.

[0016] Game server 120 can be configured to receive requests for game data from client device 110 (e.g., via Remote Procedure Call (RPC)) and respond to such requests via network 150. For example, game server 120 can encode game data in one or more data files and provide the data files to client device 110. Additionally, game server 120 can be configured to receive game data (e.g., player position, player actions, player input, etc.) from client device 110 via network 150. For example, client device 110 can be configured to periodically send game data used by game server 120 to update game data in game database 130 to reflect any and all changes in game conditions, as well as other updates, to game server 120. Various embodiments of game server 120 are described in more detail below with reference to FIG3.

[0017] Game server 120 may include or communicate with a game database 130. Game database 130 stores game data for use in parallel reality games to be served via network 150 or provided to (a number of) user devices 110. The game data stored in game database 130 may include: (1) data associated with the virtual world in the parallel reality game (e.g., image data for displaying the virtual world on a display device, geographical coordinates of the location in the virtual world, etc.); (2) data associated with players in the parallel reality game (e.g., player profiles, including (but not limited to) player information, player experience level, player currency, current player location in the virtual / real world, player energy level, player preferences, team information, faction information, etc.); (3) data associated with game objectives (e.g., data associated with the current game objective, game objective status, past game objectives, future game objectives, desired game objectives, etc.); (4) data associated with virtual elements in the virtual world (e.g., virtual elements' (5) Information related to the location of real-world objects, landmarks, and locations linked to virtual-world objects (e.g., the location of real-world objects / landmarks, descriptions of real-world objects / landmarks, and the correlation of virtual-world objects linked to real-world objects); (6) Game status (e.g., the current number of players, the current status of game objectives, player leaderboards, etc.); (7) Information related to player actions / inputs (e.g., current player location, past player location, player movement, player input, player queries, player communication, etc.); and (8) Any other information used, related to, or obtained during the implementation of the parallel reality game. The game data stored in the game database 130 may be populated offline or in real-time by the system administrator or by data received from the users / players of the game (e.g., from a client device 110 via network 150).

[0018] The feature matching system 140 matches features of an image depicting a scene with features of other images depicting the scene (or with features of a 3D model of the scene) for 3D repositioning or multi-view reconstruction. Images (or 3D models) can be received from a user device 110, a game server 120, a game database 130, or a third-party system. Images are often associated with perspective distortion due to the position and orientation of the camera capturing the image. Perspective distortion is a visual effect observed in an image where objects appear smaller as the distance between an object and the camera increases or when an object is angled toward the viewer. Since different images of the same scene can be captured by cameras with different positions and orientations, large perspective shifts can exist between images. In addition, different parts of the same image can have different perspective distortions due to differences in the position of the camera relative to different locations in the scene. The feature matching system 140 corrects the perspective distortion in the image before extracting features from it so that the extracted features are perspective-corrected features. Because perspective distortion is removed, feature matching using the extracted features is effective regardless of large perspective changes.

[0019] The feature matching system 140 removes perspective distortion from an image by using a depth map of the image. In some embodiments, the feature matching system 140 generates a depth map using a model trained to receive and output depth maps of the image. The feature matching system 140 can determine a surface normal vector for each pixel. The surface normal vector of a pixel is a vector perpendicular to the tangent plane of the surface at the pixel.

[0020] In one embodiment, the feature matching system 140 groups pixels into clusters based on the surface normal vectors of the pixels (e.g., by using k-means clustering). Each cluster is associated with a cluster surface normal vector. This cluster surface normal vector may be the surface normal vector of the cluster centroid. Since each pixel in the depth map corresponds to an element of the image (e.g., a pixel), the feature matching system 140 can also classify the corresponding pixels of the image into clusters. The feature matching system 140 generates patches from the image based on the clusters. Each patch is a part of the image, such as a region of connected pixels in the image. Since pixels with the same surface normal are located on the same plane, the patch corresponds to a plane in the scene (e.g., a building wall, ground, sky, object surface, etc.).

[0021] Additionally, the feature matching system 140 can correct the perspective of each patch (e.g., by using a homography). The feature matching system 140 calculates a homography and applies the homography to the patch. The homography performs a projection transformation and removes perspective distortion from the patch. The homography transforms the patch into a corrected patch. The corrected patch has one vertical view of the contents of the patch. The feature matching system 140 extracts features from the corrected patch. In some embodiments, the image may include one or more non-planar portions, i.e. portions not included in any of the patches. The feature matching system combines the patch and the non-planar portions (if present) to generate a perspective-corrected image and extracts features from the perspective-corrected image.

[0022] In one use case, the feature matching system 140 matches extracted features with features of another image or features of a 3D model of the same scene depicted in the image. In various embodiments, the feature matching system 140 applies a matching model to generate a mapping indicating how extracted features of an image are translated onto features of another image or onto a component of a 3D model of a scene. This mapping can be used to provide augmented reality content. For example, for a user device 110 with a camera capturing an overlapping portion of a scene, the matching model can be used to obtain a greater understanding of one or more portions of the scene captured by the camera, enabling augmented reality content (e.g., an AR character) to interact with the scene in a more realistic manner.

[0023] In some embodiments, the feature matching system 140 may be part of a game server 120 or a client device 110. Various embodiments of the feature matching system 140 are described in more detail below with reference to FIG4.

[0024] Network 150 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof. The network may also include a direct connection between a client device 110 and a game server 120. Generally, communication between the game server 120 and a client device 110 can be implemented via a network interface using any type of wired or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encoding or format (e.g., HTML, XML, JSON), or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0025] This document discusses technical reference servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from these systems. Those skilled in the art will recognize that the inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and divisions of tasks and functions among components. For example, the server programs discussed herein can be implemented using a single server or multiple servers working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel. Furthermore, the network computing environment 100 may contain different or additional components, and functionality may be distributed among the user terminal 110, server 120, database 130, and feature matching system 140 in a manner different from that described.

[0026] In situations where the systems and methods discussed herein access and analyze personal information about a user, or use personal information (such as location information), the user has the opportunity to control whether a program or feature collects information and whether or how content is received from the system or other applications. No information or data will be collected or used until the user has been provided with meaningful notification about what information will be collected and how it will be used. Information will not be collected or used unless the user provides consent that can be cancelled or modified by the user at any time. Therefore, the user can control how information about themselves is collected and how the application or system uses the information. Furthermore, before storing or using certain information or data, it can be processed in one or more ways to remove personally identifiable information. For example, a user's identity can be processed so that no personally identifiable information can be used to determine the user.

[0027] FIG2 is a block diagram of a user terminal device 120 of FIG1 according to an embodiment. In the embodiment shown in the figure, the user terminal device 120 includes a game module 210, a positioning module 220, a camera assembly 230, and an image matching module 240. In other embodiments, a user terminal device 120 may include different or additional components. Furthermore, functionality may be distributed among the components in a manner different from that described.

[0028] Game module 210 provides an interface for a player to participate in a parallel reality game. Game server 120 transmits game data via network 150 to client device 110 for use by game module 210 to provide a native version of the game to a player at a remote location on game server 120. In one embodiment, game module 210 presents a user interface on a display device associated with client device 110 that displays a virtual world associated with the game (e.g., an image of the virtual world) and allows a user to interact in the virtual world to perform various game objectives. In some embodiments, game module 210 presents image data from the real world (e.g., captured by camera assembly 230) augmented using virtual elements from the parallel reality game. In these embodiments, game module 210 may generate or adjust virtual content based on other information received from other components of client device 110. For example, game module 210 can adjust a virtual object to be displayed on the user interface based on a depth map of a scene captured from video data (e.g., generated by a depth estimation model).

[0029] Game module 210 can also control various other outputs to allow a player to interact with the game without the player looking at a display screen. For example, game module 210 can control various audio, vibration, or other notifications that allow the player to play the game without looking at a display screen. Game module 210 can access game data received from game server 120 to provide an accurate representation of the game to the user. Game module 210 can receive and process player input and provide updates to game server 120 via network 150. Game module 210 can also generate or adjust game content to be displayed by user device 110. For example, game module 210 can be based on depth information (e.g., determined by the application from one or more images captured by camera assembly 230 using a depth estimation model).

[0030] The positioning module 220 can be any device or circuit system used to monitor the location of the user terminal device 110. For example, the positioning module 220 can determine its real or relative location by using a satellite navigation positioning system (such as a GPS system, a Galileo positioning system, a Global Navigation Satellite System (GLONASS), or a BeiDou Navigation Satellite System), an inertial navigation system, an IP address-based inference navigation system, triangulation or proximity to a cell tower or Wi-Fi hotspot, or other suitable technologies for determining location. The positioning module 220 may further include various other sensors that can help accurately locate the position of the user terminal device 110.

[0031] When a player moves with the client device 110 in the real world, the positioning module 220 tracks the player's location and provides the player's location information to the game module 210. The game module 210 updates the player's location in the virtual world associated with the game based on the player's real-world location. Therefore, a player can interact with the virtual world simply by carrying or transporting the client device 110 in the real world. Specifically, the player's location in the virtual world can correspond to the player's location in the real world. The game module 210 can provide location information to the game server 120 via the network 150. In response, the game server 120 can use various techniques to verify the location of the client device 110 to prevent fraudsters from deceiving the location of the client device 110. It should be understood that location information associated with a player is only used when permission is granted after the player has been notified that access to the player's location information is permitted and how the location information can be used in the context of the game (e.g., updating the player's location in the virtual world). In addition, any location information associated with players will be stored and maintained in a manner that protects player privacy.

[0032] The camera assembly 230 includes one or more cameras configured to capture images of the physical environment in which the user device 110 is located. A portion of the physical environment depicted in an image refers to a scene. The camera assembly 230 captures image data corresponding to the scene and stores it in a storage medium on the user device. Alternatively, the image data may be transmitted to the game server 120 via the network 150. The camera assembly 230 can vary the capture rate using various variable light sensors with varying color capture ranges. The camera assembly 230 may include a wide-angle lens or a telephoto lens. The camera assembly 230 can be configured to capture a single image or video as image data. The user device 110 may also include other sensors for recording data about the physical environment of the user device, such as accelerometers, gyroscopes, inertial measurement units (IMUs), barometers, thermometers, light sensors, microphones, etc. Image data may be accompanied by additional post-data describing other details of the image data, including sensory data (such as temperature, ambient brightness) or captured data (such as exposure, temperature, shutter speed, focal length, capture time, etc.).

[0033] The image matching module 240 receives images captured by the camera assembly 230 and performs feature matching on the images. In some embodiments, the image matching module 240 sends the captured images to the feature matching system 140 for feature extraction and matching. In other embodiments, the image matching module 240 performs feature extraction and matching by using the above-described method in conjunction with the feature matching system 140.

[0034] In a configuration where the user terminal device 120 has two or more cameras, the image matching module 240 can also perform stereo matching to learn more about the scene depicted in the images captured by the cameras. In one embodiment, the image matching module 240 acquires a pair of stereo images and determines the per-pixel horizontal displacement (i.e., aberration) between pixels in the left image and the right image, or vice versa. Because the two images are captured by a camera of a single device, the camera's pose relative to each other is known. Alternatively, the stereo pair may be a pair of frames in a video captured by a single camera.

[0035] In various embodiments, the image matching module 240 applies a stereo matching model to generate a mapping indicating how pixels of one image are translated to pixels of another image in the pair. This mapping can be used to provide augmented reality content. For example, for a user device 110 having two or more images capturing overlapping portions of a scene (thereby capturing images from one or more cameras having known or estimated positions relative to each other), the stereo matching model can be used to obtain portions of the scene(s) captured by the two or more cameras such that augmented reality content (e.g., an AR character) interacts with the scene in a more realistic manner. Furthermore, the user device 110 may include a depth estimation model (e.g., a machine learning model used by the depth estimation module 410) for estimating the distance between the camera assembly 230 and entities corresponding to pixels in an image.

[0036] Figure 3 is a block diagram of a game server 120 of Figure 1 according to one embodiment. In the embodiment shown in the figure, the game server 120 includes a general game module 310, a commercial game module 320, a data collection module 330, and an event module 340. As mentioned above, the game server 120 interacts with a game database 130 that can be partially or remotely accessed by the game server 120 (e.g., the game database 130 may be a distributed database accessed via a network 150). In other embodiments, the game server 120 includes different or additional components. Furthermore, functionality may be distributed among the components in a manner different from that described. For example, the game database 130 may be integrated into the game server 120.

[0037] The Universal Game Module 310 manages the parallel reality game for all players and acts as the authoritative source of the current state of the parallel reality game for all players. As a host, the Universal Game Module 310 generates game content to be presented to players via its individual client devices 110. The Universal Game Module 310 can access the game database 130 to retrieve or store game data while managing the parallel reality game. The Universal Game Module 310 also receives game data (such as depth information, player input, player location, player actions, landmark information, etc.) from the client devices 110 and incorporates the received game data into the overall parallel reality game for all players in the parallel reality game. The Universal Game Module 310 can also manage the transmission of game data to the client devices 110 via the network 150. The general-purpose game module 310 can also manage the security status of the client device 110, including (but not limited to) ensuring the connection between the client device 110 and the game server 120, establishing connections between various client devices 110, and verifying the location of various client devices 110.

[0038] In embodiments that include a commercial game module 320, the commercial game module 320 may be separate from or part of the general game module 310. The commercial game module 320 can manage the inclusion of various game features in a parallel reality game that are linked to a real-world business activity. For example, the commercial game module 320 may receive requests via network 150 (via a network interface) from external systems (such as sponsors / advertisers, businesses, or other entities) to include game features linked to a business activity in the parallel reality game. The commercial game module 320 can then be configured to include these game features in the parallel reality game.

[0039] In embodiments including a data collection module 330, the data collection module 330 may be separate from or part of the general game module 310. The data collection module 330 can manage various game features within the parallel reality game that are linked to a data collection activity in the real world. For example, the data collection module 330 can modify game data stored in the game database 130 to include game features linked to data collection activities in the parallel reality game. The data collection module 330 can also analyze data collected by players (e.g., as part of a crowdsourcing effort) and make the data accessible on various platforms. As a concrete example, players may be prompted to submit photographs of landmarks and other features of interest in their environment, and the data collection module 330 may incorporate virtual elements corresponding to real-world landmarks or features into the parallel reality game based on the player's submission (e.g., verified landmark existence and location at the indicated submitting player).

[0040] Event Module 340 manages player access to events in a parallel reality game. Although the term "event" is used for convenience, it should be understood that this term does not necessarily refer to a specific event at a specific location or time. Rather, an "event" can refer to any access control game content in which one or more access criteria are used to determine whether a player can access that content. This content can be game content with few or no access controls or can be a part of a larger parallel reality game that is independent and has access control.

[0041] FIG4 is a block diagram of the feature matching system 140 of FIG1 according to one embodiment. In the embodiment shown in the figure, the feature matching system 140 includes a depth map module 410, a surface normal module 420, a cluster module 430, a correction module 440, and a feature module 450. In other embodiments, the feature matching system 140 includes different or additional elements. Furthermore, functionality may be distributed among the elements in a manner different from that described.

[0042] Feature matching system 140 extracts perspective correction features from an image and performs feature matching using the perspective correction features. Figures 5A to E illustrate an example pipeline for extracting perspective correction features from an image 510 performed by one embodiment of feature matching system 140. In the embodiment shown in Figure 4, feature matching system 400 includes a depth map module 410, a surface normal module 420, a cluster module 430, a correction module 440, and a feature module 450. In other embodiments, feature matching system 400 includes different or additional components. Furthermore, functionality may be distributed among the components in a manner different from that described.

[0043] The depth map module 410 receives the image 510 shown in FIG. 5A and generates a depth map 520 shown in FIG. 5B for the image 510. The image 510 depicts a scene. The scene may contain various objects, such as buildings, streets, people, vehicles, sky, etc. An object in the scene may not be entirely in the image 510 or may be partially captured by the image 510. The depth map 520 of the image 510 contains pixels, where each pixel indicates a depth value (e.g., a value in meters) of an element (e.g., a pixel) of the image. FIG. 5B includes a measuring rod 525 indicating a depth value in the range of 0 meters to 100 meters. In some embodiments, the depth map module 410 uses a model to generate the depth map 520. The model is trained using machine learning techniques based on a training dataset. The training dataset includes the image and a corresponding depth map (e.g., determined using a stereo camera, LiDAR, or other ground truth depth determination techniques). The model is trained to receive an image and output a depth map of the image. One instance of the model is a neural network, such as a convolutional neural network. The depth map module 410 extracts the depth map 520 from one of the output layers of the neural network.

[0044] The surface normal module 420 determines a surface normal vector for each (or a subset of) pixel in the depth map 520. In some embodiments, the surface normal module 420 converts the depth map 520 into a point cloud by back-projecting pixels in the depth map 520. Each pixel is represented by a point in the point cloud. In one embodiment, the surface normal module 420 applies a window (such as a 5×5 window) centered on a pixel. The surface normal module 420 fits a plane to the back-projection point corresponding to the window. For a 5×5 window, there are 25 back-projection points. The surface normal vector module 420 uses the unit normal vector of the plane as the surface normal vector of the pixel.

[0045] The cluster module 430 classifies pixels of the depth map 520 into clusters based on the surface normal vectors of the pixels. Each cluster contains a subset of pixels. In some embodiments, the cluster module 430 classifies the surface normal vectors of pixels by using k-means clustering, since all points on the same plane in the point cloud share the same or similar surface normal vectors. The cluster module 430 classifies a pixel into a cluster with the nearest cluster centroid. A cluster may contain its pairs of poles. For example, the surface normal vectors of pixels corresponding to two opposite walls are classified into the same cluster. The cluster module 430 classifies pixels based on their surface normal vectors by assigning each pixel to a cluster corresponding to its surface normal vector.

[0046] Each cluster of surface normal vectors can represent a dominant direction in the point cloud. Each cluster of pixels can represent a dominant plane in the scene, such as the ground, a building wall, the sky, etc. In one embodiment, the cluster module 430 classifies pixels into three clusters. The three clusters correspond to three planes that are orthogonal to each other in a three-dimensional space.

[0047] The cluster module 430 uses clusters to segment the image 510 into patches to identify planes in the image 510. Each patch is a region of connected pixels in the image 510. Each patch corresponds to a cluster and contains pixels in the image 510 that correspond to pixels in the depth map 520 within the cluster. As shown in Figure 5C, the image 510 is segmented into five patches 530A to 530E. Patch 530A corresponds to one side of a building. Patch 530B corresponds to the other side of a building. Patch 530C corresponds to the ground. Patches 530D and 530E correspond to two corners of a building, respectively.

[0048] In Figure 5C, the patches generated from individual clusters are represented by different patterns. Figure 5C shows patches with three different patterns: patch 530A has a dashed pattern, patch 530B has a grid pattern, and patches 530C to 530E have diagonal patterns. This indicates that the cluster module 430 has classified the pixels of the depth map 520 into three clusters representing three dominant surfaces in the scene, including two building walls and the ground. For the cluster corresponding to the ground, the cluster module 430 generates three patches 530C to 530E. In some embodiments, the cluster module 430 may identify multiple regions of connected pixels for a cluster.

[0049] The correction module 440 corrects the viewpoint of each patch based on the surface normal vectors of pixels in the corresponding cluster. In some embodiments, the correction module 440 calculates the homography of each patch to remove perspective distortion from the patch. When applied to a patch, the homography performs a projection transformation and outputs a corrected patch (i.e., a patch with corrected perspective). The homography simulates a virtual camera with the same camera center / position as the camera capturing the image but with an optical axis along the surface normal of the patch (i.e., as if the virtual camera were facing the patch). The surface normal of the patch may be the surface normal of a pixel in the corresponding cluster or the average of the surface normal vectors of pixels in the corresponding cluster. The corrected patch is a patch, as seen in the virtual camera. Each of the corrected patches 540A, 540B, and 540C has a vertical view of a dominant plane in the image.

[0050] In some embodiments, the correction module 440 corrects a portion of a patch rather than the entire patch. For a patch having a plane located at a grazing angle of a camera capturing image 510, if the entire patch is corrected, some portions of the corrected patch will be occupied by distorted or stretched areas. The correction module 440 determines whether an angular difference between an optical axis of the camera and an estimated surface normal of the patch is greater than 80 degrees. In response to determining that the angular difference is greater than 80 degrees, the correction module 440 identifies a portion of the patch such that an angular difference between the optical axis of the camera and a surface normal of the identified portion of the patch is no greater than 80 degrees. The correction module 440 corrects the identified portion of the patch but not the remaining portion of the patch.

[0051] In some embodiments, the correction module 440 determines that any part of a patch does not meet the angle difference requirement (i.e., the angular difference between the optical axis of the camera and the estimated surface normal is no greater than 80 degrees). The correction module 440 does not correct any part of the patch. Figure 5D shows three corrected patches 540A to 540C generated by the correction module 440. Corrected patches 540A to 540C correspond to patches 530A to 530C, respectively. For patches 540D and 540E, the correction module 440 determines that any part of patches 540D and 540E does not meet the angle difference requirement and does not correct any part of patches 540D and 540E.

[0052] Feature module 450 extracts features from the corrected patches 540A to 540C. A feature is a piece of information about the content of image 510, such as information indicating whether a specific region of image 510 has a specific property. A feature can be a specific component in the image, such as a point, edge, or object. A feature can also be the result of general neighborhood operations or feature detection applied to image 510. Feature module 450 can use various algorithms to detect features. Instantiated feature detection algorithms include Scale Invariant Feature Transform (SIFT), Superpoint, Binary Robust Independent Fundamental Features (BRISK), or Oriented Fast and Rotated BRIEF (ORB). When applied to image 510, the feature detection algorithm outputs a feature descriptor. A feature descriptor encodes the information of the corresponding feature into a series of numbers. The feature descriptor can act as a fingerprint of the feature and can be used to distinguish the feature from other features.

[0053] In some embodiments, the corrected patches 540A to 540E may not cover the entire image 510. The image may have a non-planar portion (i.e., a portion not included in any of the patches generated by the clustering module 430). The feature module 450 identifies the non-planar portion (or several non-planar portions) in the image 510 (e.g., by comparing patches 530A to 530E with the image 510). In some embodiments, the feature module 450 combines the non-planar portion with the patch and generates a perspective-corrected image. The feature module 450 extracts features from the perspective-corrected image. In some other embodiments, the feature module 450 extracts features from the non-planar portion and the corrected patch, respectively. Then, the feature module 450 determines the two-dimensional position of the features extracted from the corrected patch and wraps the features back into a coordinate system of the image 510 without changing the feature descriptor.

[0054] Feature module 450 (for example) matches the extracted features with features of a 3D model of another image or scene by using a descriptor of the extracted features. Feature matching can be a step in Simultaneous Localization and Mapping (SLAM), Structure for Motion Inference (SfM), visual localization, or other computer vision procedures. For example, in the context of a parallel reality game, feature matching can be used for 3D relocalization (e.g., relocalizing a camera relative to a 3D map of the surrounding area). It can also be used for multi-view reconstruction of a scene.

[0055] In some embodiments, feature module 450 performs feature matching on image pairs in a dataset using perspective correction features. The dataset contains images depicting a plurality of scenes. The images may originate from a continuous video sequence (e.g., a video sequence captured by a user terminal device 110). Each scene has several images captured from different viewpoints, under different environmental conditions, at different locations, under different weather conditions, or some combination thereof. Feature module 450 identifies a plurality of image pairs for each scene. Feature module 450 may use perspective correction features to establish two-dimensional or three-dimensional (i.e., stereo) matching. Feature module 450 determines a relative camera pose and compares the relative camera pose with the relative camera pose of a ground truth of one of the scenes to locate the image (and thus the camera capturing the image).

[0056] In some embodiments, the feature module 450 inputs perspective correction features into a matching model to generate a mapping indicating how features of one image are translated onto features of the other image in the pair. This mapping can be used to provide augmented reality content. The mapping between image pairs of a scene can be used to obtain an understanding of the scene in which the user device 110 is located (such as the 3D position and orientation of the user device 110 relative to a 3D map of the surrounding scene), regardless of the scene's weather conditions, environmental conditions, etc., which allows an AR character to interact with the scene in a realistic manner.

[0057] Exemplary Parallel Reality Game

[0058] Figure 6 depicts a conceptual diagram of a virtual world 610 parallel to a real world 600, which is a game board that can act as a parallel reality game for a player, according to one embodiment. As shown in the figure, the virtual world 610 may include a geography parallel to the geography of the real world 600. Specifically, a coordinate range defining a geographical region or space in the real world 600 is mapped to a corresponding coordinate range defining a virtual space in the virtual world 610. The coordinate range in the real world 600 may be associated with a town, neighborhood, city, campus, place, country, continent, the entire Earth, or other geographical region. Each geographical coordinate within the geographical coordinate range is mapped to a corresponding coordinate in a virtual space in the virtual world.

[0059] The location of a player in the virtual world 610 corresponds to the location of a player in the real world 600. For example, player A, located at location 612 in the real world 600, has a corresponding location 622 in the virtual world 610. Similarly, player B, located at location 614 in the real world, has a corresponding location 624 in the virtual world. When a player moves within a geographic coordinate range in the real world, the player also moves within a coordinate range defining the virtual space in the virtual world 610. Specifically, when a player navigates a geographic coordinate range in the real world, a positioning system (e.g., a GPS system) associated with a mobile computing device carried by the player can be used to track a player's location. Data associated with the player's location in the real world 600 is used to update the player's location within the corresponding coordinate range defining the virtual space in the virtual world 610. In this way, players can navigate along a continuous trajectory within the coordinates of the virtual space defined in the virtual world 610 simply by moving within the corresponding geographical coordinates in the real world 600, without registering or periodically updating their location information at specific discrete locations in the real world 600.

[0060] Location-based games may contain multiple game objectives that require a player to travel to or interact with various virtual elements or objects at various virtual locations in a virtual world. A player can travel to such virtual locations by moving to the corresponding location of a virtual element or object in the real world. For example, a positioning system can continuously track the player's location so that as the player continuously navigates the real world, the player also continuously navigates a parallel virtual world. The player can then interact with various virtual elements or objects at specific locations to achieve or perform one or more game objectives.

[0061] For example, a game objective is for a player to interact with virtual elements 630 at various virtual locations in a virtual world 610. These virtual elements 630 may be linked to landmarks, geographical locations, or objects 640 in the real world 600. Real-world landmarks or objects 640 may be works of art, monuments, buildings, businesses, libraries, museums, or other suitable real-world landmarks or objects. Interactions include acquiring, claiming ownership, using virtual items, spending virtual currency, etc. To acquire such virtual elements 630, a player must travel to a landmark or geographical location 640 linked to the virtual element 630 in the real world and must perform any necessary interactions with the virtual element 630 in the virtual world 610. For example, player A may have to travel to a landmark 640 in the real world 600 to interact with or acquire a virtual element 630 linked to that particular landmark 640. Interaction with the virtual element 630 may require action in the real world, such as taking a photo or verifying, obtaining or retrieving other information about the landmark or object 640 associated with the virtual element 630.

[0062] Game objectives may require players to use one or more virtual items collected by the player in a location-based game. For example, a player may move through virtual world 610 to find virtual items (such as weapons, creatures, power-on items, or other items) that can be used to complete game objectives. These virtual items may be found or collected by moving to different locations in real world 600 or by performing various actions in virtual world 610 or real world 600. In the example shown in Figure 6, a player uses virtual item 632 to retrieve one or more virtual elements 630. Specifically, a player may approach virtual element 630 or deploy virtual item 632 at a location within virtual world 610 within virtual element 630. Deploying one or more virtual items 632 in this manner may result in retrieving virtual elements 630 belonging to a specific player or a specific player's team / faction.

[0063] In one particular implementation, a player may be required to collect virtual energy as part of a parallel reality game. As depicted in Figure 6, virtual energy 660 can be scattered at different locations in the virtual world 610. A player can collect virtual energy 660 by moving to the corresponding location of virtual energy 660 in the real world 600. Virtual energy 660 can be used to power virtual items or to perform various game objectives. Losing one of the virtual energies 660 disconnects the player from the game.

[0064] According to the present invention, a parallel reality game can be a game based on a large number of multiplayer locations, in which each participant in the game shares the same virtual world. Players can be divided into separate teams or factions and can work together to achieve one or more game objectives, such as acquiring or claiming ownership of a virtual element. In this way, a parallel reality game can essentially be a social game that encourages cooperation among players within the game. During a parallel reality game, players from opposing teams can compete against each other (or sometimes cooperate to achieve mutual objectives). A player can use virtual items to attack or hinder the progress of players in the opposing team. In some cases, players are encouraged to gather in real-world locations to cooperate or interact with events in the parallel reality game. In these cases, the game server seeks to ensure that players are actually present and not deceiving.

[0065] Parallel reality games may have various features to enhance and encourage gameplay within the game. For example, players may accumulate virtual currency or another virtual reward (e.g., virtual tokens, virtual pads, virtual material resources, etc.) which can be used throughout the game (e.g., to purchase in-game items, exchange for other items, craft items, etc.). Players can advance through various levels as they complete one or more game objectives and accumulate experience within the game. In some embodiments, players may communicate with each other through one or more communication interfaces provided in the game. Players may also acquire enhanced "power" or virtual items that can be used to complete game objectives within the game. Those skilled in the art using the disclosure provided herein will understand that parallel reality games may include various other game features without departing from the scope of the invention.

[0066] Figure 7 is a flowchart illustrating one method 700 for extracting perspective correction features from an image according to an embodiment. In some embodiments, method 700 is performed by feature matching system 140, although in other embodiments, some or all of the operations in method 700 may be performed by other entities. In some embodiments, the operations in method 700 are performed in a different order and may include different or additional steps.

[0067] Feature matching system 140 receives 710 an image of a scene. Feature matching system 140 determines a depth map 720 of the image. In some embodiments, feature matching system 140 determines the depth map of an image by inputting the image into a model. The model is trained to receive images and output depth maps of the received images. In one instance, the model is trained using machine learning techniques based on a training set. The training set includes a set of images and a depth map of each image in the set. In some embodiments, the model is a convolutional neural network.

[0068] Feature matching system 140 calculates 730 a surface normal vector for each (or a subset) of the pixels in the depth map. The surface normal vector indicates an estimated surface normal of the pixel. In some embodiments, feature matching system 140 back-projects pixels in the depth map to points in a point cloud. Feature matching system 130 calculates the surface normal vectors of the pixels in the point cloud.

[0069] The feature matching system 140 classifies the pixels into clusters 740 based on the surface normal vectors of the pixels, each cluster containing a subset of the pixels. In some embodiments, the feature matching system 140 uses k-means clustering to classify pixels into clusters. In some embodiments, the feature matching system 140 classifies surface normal vectors into three clusters based on the surface normal vectors of the pixels. The three clusters correspond to three planes orthogonal to each other in a three-dimensional space.

[0070] The feature matching system 140 generates 750 patches from the image based on the clusters. Regions of connected pixels in the image correspond to at least some pixels in the depth map of one of the clusters. In some embodiments, the feature matching system 140 generates multiple patches for a cluster.

[0071] The feature matching system 140 corrects the perspective distortion of one of the 760 individual patches based on the surface normals in the corresponding cluster to produce a corrected patch. In some embodiments, the feature matching system 140 computes homography of a virtual camera having the same camera center as a camera capturing an image and having an optical axis along an estimated surface normal of the corresponding cluster. In one embodiment, the estimated surface normal of the corresponding cluster is the average of the surface normals in the corresponding cluster. The feature matching system 140 uses homography to perform a projection transformation to transform the patch into a corrected patch.

[0072] In some embodiments, the feature matching system 140 determines whether an angular difference between an optical axis of a camera and a surface normal of a patch is greater than a threshold value, such as 80 degrees. In response to determining that the angular difference between the optical axis of the camera and the surface normal of the patch is greater than 80 degrees, the feature matching system 140 identifies a portion of the patch. If the angular difference between the optical axis of the camera and the surface normal of the identified portion of the patch is not greater than 80 degrees, the feature matching system 140 corrects the identified portion of the patch without correcting the remaining portion of the patch.

[0073] Feature matching system 140 extracts 770 perspective correction features from the corrected patches. Feature matching system 140 matches the perspective correction features with features of another image to reposition an object in the scene. In some embodiments, feature matching system 140 identifies planar and non-planar portions of the image. Feature matching system 140 generates a perspective-corrected image and extracts corrected patches from the planar portions of the image. Feature matching system 140 extracts features from the corrected patches. In some embodiments, feature matching system 140 also extracts features from the non-planar portions of the image. The final set of features used for feature matching includes perspective correction features and features extracted from the non-planar portions of the image.

[0074] Instance-based computing system

[0075] FIG8 is a block diagram illustrating a functional view of a typical computer system 800, which serves as a user terminal device 110, server 120, or feature matching system 140 of FIG1, according to an embodiment. Although FIG8 depicts various physical components of a computer that are part or all of one or more of the entities described herein, a computer may have additional, fewer, or varied components shown in FIG8. As will be appreciated by those skilled in the art, items shown separately may be combined and some items may be divided into multiple parts.

[0076] The system illustrated in FIG8 is coupled to at least one processor 802 of a chipset 804. Also coupled to the chipset 804 are a memory 806, a storage device 808, a keyboard 810, a graphics adapter 812, a pointing device 814, and a network adapter 816. A display 818 is coupled to the graphics adapter 812. In one embodiment, the functionality of the chipset 804 is provided by a memory controller hub 820 and an I / O hub 822. In another embodiment, the memory 806 is directly coupled to the processor 802 instead of the chipset 804. In some embodiments, the computer 800 includes one or more communication buses for interconnecting these components. The one or more communication buses may include circuitry (sometimes referred to as a chipset) for interconnecting and controlling communication between system components.

[0077] Storage device 808 is any non-transitory computer-readable storage medium, such as a hard disk drive, optical disc read-only memory (CD-ROM), DVD, or a solid-state insulating device or other optical storage, magnetic card, magnetic tape, magnetic disk storage or other magnetic storage device, optical disc storage device, flash memory device or other non-volatile solid-state storage device. This storage device 808 may also refer to persistent memory. Pointer 814 may be a mouse, trackball, or other type of pointer device, and is used in conjunction with keyboard 810 to input data into computer 800. Graphics adapter 812 displays images and other information on monitor 818. Network adapter 816 couples computer 800 to a local area network or wide area network.

[0078] Memory 806 holds instructions and data used by processor 802. Memory 806 may be non-persistent memory, examples of which include high-speed random access memory, such as DRAM, SRAM, DDR-RAM, ROM, EEPROM, and flash memory.

[0079] As is known in the art, computer 800 may have components other than those shown in FIG8. Additionally, computer 800 may lack certain graphics components. In one embodiment, computer 800 acting as a server may lack a keyboard 810, pointing device 814, graphics adapter 812, or display 818. Furthermore, storage device 808 may be local or remote to computer 800 (e.g., embodied in a storage area network (SAN)).

[0080] As known in the art, computer 800 is adapted to execute a computer program module for providing the functionality described herein. As used herein, the term "module" refers to computer program logic for providing the specified functionality. Thus, a module may be implemented in hardware, firmware, or software. In one embodiment, the program module is stored on storage device 808, loaded into memory 806, and executed by processor 802.

[0081] Additional considerations

[0082] Some of the foregoing descriptions describe embodiments of algorithmic programs or operations. These algorithmic descriptions and representations are commonly used by those skilled in the fields of data processing and computer vision to effectively communicate the essence of their work to other skilled articulators. Although described functionally, operationally, or logically, these operations should be understood as being implemented by a computer program, including instructions for execution by a processor or equivalent circuitry, microcode, or the like. Furthermore, referring to such configurations of functional operations as modules is sometimes convenient without losing generality.

[0083] As used herein, any reference to "an embodiment" is intended to include, together with the specific elements, features, structures, or characteristics described in the embodiment, in at least one embodiment. The appearance of the phrase "in an embodiment" in various places in the specification does not necessarily refer to the same embodiment.

[0084] The terms "coupled" and "connected," along with their derivatives, may be used to describe some embodiments. It should be understood that these terms are not intended to be synonymous with each other. For example, the term "connected" may be used to describe some embodiments to indicate that two or more elements are in direct physical or electrical contact with each other. In another instance, the term "coupled" may be used to describe some embodiments to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. In this context, the embodiments are not limited thereto.

[0085] As used herein, the terms "comprising," "including," "having," or any other variation thereof are intended to cover a non-exclusive inclusion. For example, a procedure, method, article, or apparatus that includes one of the listed elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to that procedure, method, article, or apparatus. Furthermore, unless expressly stated otherwise, "or" means an inclusive or rather than an exclusive or. For example, a condition A or B is satisfied by either: A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); and both A and B are true (or exist).

[0086] In addition, the term "a" is used to describe the elements and components of the embodiments. This is done for convenience only and to give the general meaning of the invention. Unless it is obvious otherwise, this specification should be understood to include "a" or at least "a", and the singular includes the plural.

[0087] Upon reading this invention, those skilled in the art will understand that an alternative structure and functional design of a system and a procedure for verifying an account and an online service provider corresponds to a real business. Therefore, although specific embodiments and applications have been illustrated and described, it should be understood that the described subject matter is not limited to the precise construction and components disclosed herein, and various modifications, alterations, and variations will be readily apparent to those skilled in the art in the configuration, operation, and details of the disclosed methods and apparatus. The scope of protection should be limited only to the following claims.

Claims

1. A computer-implemented method, comprising: The system receives an image of a scene; determines a depth map of the image, the depth map containing a plurality of pixels; for each of the pixels, calculates a surface normal vector indicating an estimated surface normal of the pixel; classifies the pixels in the depth map into clusters based on the surface normal vectors of the pixels, each cluster containing a subset of the pixels in the depth map; generates patches from the image based on the clusters, each patch being a region of connected pixels in the image, the region of connected pixels in the image corresponding to at least some pixels in the depth map of the clusters; corrects the perspective distortion of each patch based on the surface normal vectors of the pixels in the corresponding clusters to generate a rectified patch; and extracts perspective correction features from the rectified patches, the perspective correction features being used for feature matching.

2. The computer implementation method of claim 1, wherein determining the depth map of the image includes: The depth map of the image is determined by inputting the image into a model, which is trained to receive images and output the depth map of the received images.

3. The computer implementation method of claim 2, wherein the model is a convolutional neural network.

4. The computer implementation method of claim 1, wherein classifying the pixels into clusters based on the surface normals of the pixels includes: k-means clustering is used to classify pixels into clusters based on their surface normals.

5. The computer implementation method of claim 1, wherein classifying the pixels into clusters based on the surface normal vectors of the pixels includes: Based on the surface normal vectors of these pixels, these surface normal vectors are classified into three clusters, which correspond to three mutually orthogonal planes in a three-dimensional space.

6. The computer implementation method of claim 1, wherein correcting the perspective distortion of each patch of the patches based on the surface normal vectors of the pixels in the corresponding cluster to generate the corrected patch includes: The computation simulates a homography of a virtual camera having the same camera center as a camera that captures the image and having an optical axis along an estimated surface normal of the corresponding cluster; and uses the homography to perform a projection transformation from the patch to the corrected patch.

7. The computer implementation method of claim 6, wherein the estimated surface normal of the corresponding cluster is the average of one of the surface normal vectors of the pixels in the corresponding cluster.

8. The computer implementation method of claim 1, wherein the image is captured by a camera, wherein correcting the perspective view of each patch of the patches based on the surface normal vectors of the pixels in the corresponding cluster to generate the corrected patch includes: Determine whether the angular difference between one optical axis of the camera and the surface normal of one of the patches is greater than 80 degrees; in response to determining that the angular difference between one optical axis of the camera and the estimated surface normal of one of the patches is greater than 80 degrees, identify a portion of the patch, and that the angular difference between the optical axis of the camera and the surface normal of the identified portion of the patch is not greater than 80 degrees; and correct the perspective view of the identified portion of the patch without correcting the remaining portion of the patch.

9. The computer-implemented method of claim 1, wherein extracting the perspective correction features from the corrected patches includes: Identify a non-planar portion of the image that is not included in any of the patches; extract features from the non-planar portion of the image; and extract features including perspective correction features and features extracted from the non-planar portion of the image.

10. The computer implementation method of claim 1, further comprising: These perspective correction features are matched with the features of another image to reposition the image relative to the other image.

11. A non-transitory computer-readable memory storing executable computer program instructions, the instructions being executable to perform operations including: receiving an image of a scene; determining a depth map of the image, the depth map comprising a plurality of pixels; for each of the pixels, calculating a surface normal vector indicating an estimated surface normal of the pixel; classifying the pixels in the depth map into clusters based on the surface normal vectors of the pixels, each cluster comprising a subset of the pixels in the depth map; generating patches from the image based on the clusters, each patch being a region of connected pixels in the image, the region of connected pixels in the image corresponding to at least some pixels of the pixels in the depth map of one of the clusters; correcting a perspective distortion of each patch of the patches based on the surface normal vectors of the pixels in the corresponding clusters to generate a corrected patch; and extracting perspective correction features from the corrected patches, the perspective correction features being used for feature matching.

12. The non-transitory computer-readable memory as in claim 11, wherein determining the depth map of the image includes: The depth map of the image is determined by inputting the image into a model, which is trained to receive images and output the depth map of the received images.

13. The non-transitory computer-readable memory of claim 11, wherein classifying the pixels into clusters based on the surface normals of the pixels includes: k-means clustering is used to classify pixels into clusters based on their surface normals.

14. The non-transitory computer-readable memory of claim 11, wherein classifying the pixels into clusters based on the surface normals of the pixels includes: Based on the surface normal vectors of these pixels, these surface normal vectors are classified into three clusters, which correspond to three mutually orthogonal planes in a three-dimensional space.

15. The non-transitory computer-readable memory of claim 11, wherein correcting the perspective distortion of each patch of the patches based on the surface normals of the pixels in the corresponding cluster to produce the corrected patch includes: The computational simulation has a homography of a virtual camera with the same center as a camera capturing the image and an optical axis along an estimated surface normal of the corresponding cluster; and uses the homography to perform a projection transformation from the patch to the corrected patch.

16. The non-transitory computer-readable memory of claim 11, wherein the image is captured by a camera, wherein correcting the perspective view of each patch of the patches based on the surface normals of the pixels in the corresponding cluster to produce the corrected patch includes: Determine whether the angular difference between one optical axis of the camera and the surface normal of one of the patches is greater than 80 degrees; in response to determining that the angular difference between one optical axis of the camera and the estimated surface normal of one of the patches is greater than 80 degrees, identify a portion of the patch, and that the angular difference between the optical axis of the camera and the surface normal of the identified portion of the patch is not greater than 80 degrees; and correct the perspective view of the identified portion of the patch without correcting the remaining portion of the patch.

17. The non-transitory computer-readable memory as claimed in claim 11, wherein extracting the perspective correction features from the corrected patches includes: Identify a non-planar portion of the image that is not included in any of the patches; extract features from the non-planar portion of the image; and extract features including perspective correction features and features extracted from the non-planar portion of the image.

18. The non-transitory computer-readable memory as claimed in claim 11, further comprising: These perspective correction features are matched with the features of another image to reposition the image relative to the other image.

19. A computer system comprising: A computer processor, which is used to execute computer program instructions; The system includes a non-transitory computer-readable memory storing computer program instructions executable by a computer processor to perform operations including: receiving an image of a scene; determining a depth map of the image, the depth map comprising a plurality of pixels; for each of the pixels, calculating a surface normal vector of an estimated surface normal of the pixel; classifying the pixels in the depth map into clusters based on the surface normal vectors of the pixels, each cluster comprising a subset of the pixels in the depth map; generating patches from the image based on the clusters, each patch being a region of connected pixels in the image, the region of connected pixels in the image corresponding to at least some pixels of the pixels in the depth map of one of the clusters; correcting a perspective view of each patch of the patches based on the surface normal vectors of the pixels in the corresponding clusters to generate a corrected patch; and extracting perspective correction features from the corrected patches, the perspective correction features being used for feature matching.

20. The computer system of claim 19, wherein classifying pixels into clusters based on the surface normals of the pixels includes: Based on the surface normal vectors of these pixels, these surface normal vectors are classified into three clusters, which correspond to three mutually orthogonal planes in a three-dimensional space.

Images