Camera pose estimation using fuzzy features

Abstract

A method for estimating a camera pose includes identifying a three-dimensional (3D) map representing a physical environment, the 3D map including 3D map features defined as 3D points. A blurred image representation is received, the representation being derived from an original unblurred image of the physical environment captured by a camera. The representation includes a plurality of blurred features, each comprising: (i) a two-dimensional (2D) line passing through a 2D point in the original unblurred image at which the image feature is detected; and (ii) a feature descriptor describing the image feature associated with the 2D point through which the 2D line of the blurred feature passes. A correspondence between the blurred features and 3D map features of the 3D map of the physical environment is determined. Based on the determined correspondence, a six-degree-of-freedom pose of the camera in the physical environment is estimated.

Description

Background Art

[0001] Image-based localization techniques can be utilized by mobile display devices, robots, drones, and other devices to estimate their position and orientation in a physical environment. For example, one or more images captured by a device can be compared to a pre-computed map representing a 3D reconstruction of a space. Summary of the invention

[0002] This summary is provided to introduce some concepts in a simplified form, which will be further described in the detailed description below. This summary is neither intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.

[0003] A method for estimating a camera pose includes identifying a three-dimensional (3D) map representing a physical environment, the 3D map including a plurality of 3D map features, the 3D map features being defined as 3D points distributed throughout the 3D map. A blurred image representation is received, the representation being derived from an original unblurred image of the physical environment captured by a camera. The representation includes a plurality of blurred features, each comprising: (i) a two-dimensional (2D) line passing through a 2D point at which the image feature was detected in the original unblurred image, and (ii) a feature descriptor describing an image feature associated with the 2D point through which the 2D line of the blurred feature passes. A correspondence between the blurred features and the 3D map features of the 3D map of the physical environment is determined. Based on the determined correspondence, a six-degree-of-freedom pose of the camera in the physical environment is estimated. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 Example images depicting the physical environment.

[0005] Figure 2 An example method for estimating camera pose is illustrated.

[0006] Figure 3 Example client-side devices and remote devices that may be used to estimate camera pose are schematically illustrated.

[0007] Figure 4 Depicted Figure 1 Example images of the physical environment of FIG, in which image features are detected.

[0008] Figure 5 Depicted in Figure 1 The image features detected in the image.

[0009] Fig. 6A Example image features including 2D locations are shown.

[0010] Figure 6B Shows the passage through Fig. 6A An example image of a 2D line that associates 2D points with features.

[0011] Figure 6C Shown with Fig. 6A The fuzzy features corresponding to the example image features, but using Figure 6B The 2D line is replaced by Fig. 6A The 2D position of

[0012] Figure 7 Another example method for estimating camera pose is illustrated.

[0013] Fig. 8A , Figure 8B and Figure 8C The determination of the correspondence between blurred features and 3D mapped features is illustrated.

[0014] Fig.9A and Fig. 9B The correspondence between the blurred features and the 3D mapped features is illustrated.

[0015] Fig.10 The estimation of the camera pose based on the determined correspondences is illustrated.

[0016] Figures 11A-11D Illustrated is the generation of a blurred 3D local map based on two or more images of a physical environment.

[0017] Fig.12 An example computing system is schematically illustrated. DETAILED DESCRIPTION

[0018] A wide variety of devices and related technologies can utilize image-based positioning techniques. Examples can include wearable computing devices, such as head-mounted display (HMD) devices that provide augmented reality and / or virtual reality experiences, robotic machines, and autonomous vehicles. These and other devices can use image-based positioning techniques to determine their position and orientation (i.e., pose) in a physical environment. In this document, such devices will be referred to as "camera devices." Camera devices can have any appropriate hardware and take a mobile or non-mobile form factor. The following is a description of image-based positioning techniques. Figure 1 , Figure 3 and Fig.12 Described camera device 104 , camera device 300 , and computing system 1200 are non-limiting examples of camera devices.

[0019] Some methods for image-based positioning utilize knowledge of previous three-dimensional (3D) mapping or scene models of the physical environment. A 3D map is essentially a spatial database including geometric data. In some examples, the geometric data includes points defined in three dimensions (e.g., x, y, z) distributed throughout the 3D space to form a 3D point cloud reconstruction of the scene. The 3D points are associated with feature descriptors (e.g., multidimensional vectors) extracted from source images or videos used to construct the mapping. Therefore, the 3D points can be referred to as 3D mapping features of the 3D mapping of the physical environment.

[0020] Given a 3D map of the physical environment and using one or more images or videos captured by a camera, the image-based localization task can estimate the 3D position (x, y, z) and 3D orientation (roll, pitch, yaw) of the camera in the physical environment. Camera localization algorithms can solve this task in two steps - first by matching 2D point features in the captured image with 3D mapped features stored as 3D points in the 3D map. Next, the six degrees of freedom (6DOF) pose of the camera can be calculated using the 2D point to 3D point matching and its base coordinates.

[0021] In some cases, camera positioning can be performed by a device other than the camera device used to capture the image, thereby potentially saving processing, battery, and / or storage resources of the camera device. For example, the image can be transmitted to a remote device (e.g., a server), which can access a 3D map of the physical environment and perform processing steps for estimating the camera pose. In addition, transmitting the image to the remote device alleviates the need to share the 3D map with a third party, who can infer information about the physical environment based on the 3D map. The camera pose can optionally be transmitted back to the camera device. However, in some examples, the image used to determine the camera pose may include sensitive, confidential, and / or private information related to the depicted physical environment. For example, the image may depict the user's face, sensitive items or personal items present in the scene, confidential information, etc. When the image is transmitted to the remote device, this can cause privacy issues because the user of the camera device may not want their private images to be transmitted to or stored on a remote device that is not under their control.

[0022] It is worth noting that image localization can be performed solely based on image features detected within the image. Therefore, if the image features (e.g., the 2D positions of the image features and their associated feature descriptors) are available to the remote device, the actual image content (e.g., pixel values) does not need to be transmitted to the remote device. However, in some cases, a lossy representation of the original image can be recreated based on information associated with the image features (i.e., the 2D positions of the image features detected in the image and the associated feature descriptors). In other words, upon receiving the 2D pixel positions of the image features within the image and the associated feature descriptors, the remote device can recreate an image of the user's current surroundings, thereby potentially leaking private information.

[0023] The present disclosure solves the above-mentioned confidentiality and privacy issues by blurring the image features transmitted to the remote device. Specifically, when capturing an image of a physical environment, the device can detect image features in the image and generate multiple blurred features representing the image features without revealing their associated 2D positions. Specifically, each blurred feature can include a 2D line passing through the 2D position of the detected image feature, and a feature descriptor describing the image feature. For the purpose of camera pose estimation, the blurred features can be transmitted to the remote device as part of the blurred image representation. It is worth noting that the blurred image representation does not include the 2D position of the image feature, which makes it significantly more difficult (if not impossible) for the remote device to recreate the original image. However, the blurred image representation can include sufficient information for the remote device to estimate the pose of the camera relative to the 3D mapping of the physical environment. Therefore, the privacy issues associated with transmitting potential private information to the remote device can be alleviated while retaining the advantages of using a remote device to perform camera positioning, such as saving processing, storage and battery resources of the camera device.

[0024] Figure 1 An image 100 depicts a physical environment 102. Although Figure 1 100 may be captured by a camera device 302 of a physical environment, as described below with respect to the camera device 300. Figure 3 Give a description.

[0025] Environment 102 includes structure 104, in this case, structure 104 is a famous building. As described above, an image 100 of the physical environment 102 captured by a camera device can be transmitted to a remote device (e.g., a server) for camera pose estimation. For example, the remote device may have previously acquired a 3D map of the physical environment, the 3D map including a plurality of 3D map features. The map features can be characterized as a reconstructed 3D point cloud representing the physical environment. By determining the correspondence between the map features in the 3D map and the image features detected in the image, the pose of the camera device when the image 100 was captured can be inferred.

[0026] However, the image of environment 102 also includes user 106, whose face is clearly visible in the image. Users may not want their facial images to be transmitted to a remote device that the user cannot control because this raises privacy concerns. Even in the case where the remote device is trusted or controlled by the user (e.g., a server owned by an individual or company), the image may be intercepted en route and / or illegally retrieved from the server by another party. In addition, even when the information transmitted to the server includes only image features detected in the image, and does not include the image pixel values ​​themselves, it is possible to recreate a lossy representation of the image, which may reveal the user's face.

[0027] Accordingly, Figure 2 An example method 200 for estimating camera pose that relies on blurred image representations to prevent unauthorized recreation of the original image is illustrated. The method 200 may be implemented on any suitable camera device and / or cooperating remote device. Figure 3 An example camera device is schematically shown in FIG. Figure 3 A camera device 300 and a cooperating remote device 350 are shown communicatively coupled via a network 320. Each of the camera device 300 and the remote device 350 can take any suitable form. In different examples, the camera device 300 and / or the remote device 350 can take the form of a head mounted display, a server, a networked computer, a gaming console, a mobile communication device, a desktop computer, a laptop computer, a tablet computer, a set-top box (e.g., a cable box, a satellite box), or any other type of suitable computing device. As specific examples, the camera device can be implemented as an HMD, an autonomous vehicle, an aerial drone, and / or other suitable camera device. Fig.12 Additional details regarding components and computational aspects of camera device 300 and remote device 350 are described.

[0028] At 202, Figure 2 The method 200 includes capturing an image of a physical environment. Figure 3As shown in , the camera device includes a camera 302 that can be used to capture images of the physical environment. In some examples, the camera 302 may include an RGB camera, an IR camera, and / or a depth camera (such as a time-of-flight, structured light, or other form of depth camera). The camera device 300 also includes a logic machine 304 and a storage machine 306, which holds instructions executable by the logic machine to perform various computing tasks. The logic machine 304 and the storage machine 306 can be implemented using any suitable data processing and data storage technology, respectively. Fig.12 Provides additional details about logical machines and storage machines.

[0029] exist Figure 3 , an image 308 captured by camera 302 is maintained by a storage device 306 of camera device 300 . Figure 1 Image 100 is one example of such an image, although the camera device may capture any image of any physical environment.

[0030] return Figure 2 At 204, method 200 includes detecting a plurality of image features at two-dimensional pixel locations within an image of a physical environment. Figure 3 The storage device 306 shown as the camera device 300 in FIG. 1 also includes a plurality of image features 310 associated with the image 308. Figure 4 , Figure 1 The image 100 is again shown together with black dots, which represent the two-dimensional locations where image features have been detected. One such dot corresponds to one image feature 402. It should be understood that Figure 4 Each point shown in may have an associated image feature, and an image may have any number of such features.

[0031] Reference now Figure 5 , from Figure 4 Black dots representing image features detected in image 100 are illustrated. As shown in this example, the two-dimensional distribution of image features can reflect the geometric properties and visual appearance of objects present in the physical environment when the image was taken. In this example, the image features include features corresponding to the shape, size, and other features of structure 104 and user 106.

[0032] Figure 5 Also included is a schematic representation of a single image feature 402. Depending on the embodiment, the image feature may include any appropriate information. As shown, the image feature 402 includes a feature descriptor 500, a two-dimensional position 502 (e.g., the pixel coordinates of a black dot representing the image feature within the image), a scale 504, and a one-dimensional angular orientation 506.

[0033] Image features can take various suitable forms and can be detected in various suitable ways. As an example, image features can be scale-invariant feature transform (SIFT) features. SIFT features include a SIFT key point that stores geometric information related to the feature, including the 2D position, scale, and one-dimensional angular orientation of the feature. SIFT features also include feature descriptors, which are implemented as 128-dimensional vectors. A multi-step process can be used to detect SIFT features in an image. However, it should be understood that the process is provided as an example. Any suitable method for extracting local features can be used to identify image features, and such image features do not have to be SIFT features. As another example, the detected image features can be speeded up robust features (SURF).

[0034] To identify SIFT features, multiple copies of the original image may be generated, with each new copy being progressively blurred. In some cases, once a predetermined number of blurred images have been created, a new copy of the original image may be created that has a reduced size (e.g., half the size) compared to the original image. Progressively blurred copies of the reduced size image may then be generated, and this may be repeated any number of times to give a set (or "octave") of any number of image copies, each set having a different scale. In some cases, the set of image copies may be referred to as a "scale space."

[0035] Once the scale space is generated, a Difference of Gaussian (DoG) operation can be performed to identify the differences between image copies with the same scale (i.e., in the same octave). In other words, for two blurred copies of an original image with the same scale, if one copy is more blurred than the other, a DoG image is generated indicating the difference in pixel values ​​between the two blurred copies. For example, any pixel that is the same in the two blurred images (i.e., the difference is zero) can be represented as black in the DoG image. Pixels that are different between the two blurred copies (i.e., the difference is not zero) can be represented in the DoG image using a non-black grayscale value, so that larger differences in pixel values ​​are represented as whiter than smaller differences in pixel values. This results in a set of DoG images in which areas of higher variability in the original image (e.g., corners or edges within the image) are lighter in color than areas with less variability. Any appropriate number of DoG images can be generated, depending on how many octaves of blurred image copies are generated.

[0036] Next, local maxima and minima may be identified within at least some of the DoG images. This may be accomplished by, for each pixel in a given DoG image, checking whether the pixel has the highest or lowest value relative to a certain number (e.g., 26) of neighboring pixels. Once the maximum and minimum values ​​are identified at the pixel level, a Taylor expansion may be applied to the maximum and minimum values ​​within the DoG image to find the maximum and minimum values ​​at the sub-pixel level.

[0037] At least some of the identified maximum and minimum values ​​may have relatively low contrast in the original image, be located along the edge, and / or be generally unsuitable as image features. Therefore, some number of maximum and minimum values ​​may be excluded from being SIFT features. This can be done by first excluding any pixels having less than a threshold intensity, which can be set to any appropriate value according to the total expected number and uniqueness of the final feature. Next, for each potential feature, two vertical gradients can be calculated near the pixel of the potential feature and subtracted from each other. If the potential feature is far away from the edge or corner, the two gradients may be small and the difference will be close to zero. Similarly, if the potential feature is located at a corner, both gradients will be large and the difference will be close to zero again. Potential features placed along the edge will have a larger gradient and a smaller gradient, and the difference is larger. Because features located along the edge are generally not as expected as corner features, edge features can be discarded. In some cases, a Harris corner detector can be used to calculate corner features.

[0038] Once the final set of SIFT features is identified, keypoint orientations and feature descriptors can be computed. Keypoint orientations can be computed by evaluating the gradient direction and magnitude near the pixel of the feature, which can be collected in a histogram with a set number of bins (e.g., each bin corresponds to an angle range). The bin with the highest magnitude can be set as the keypoint orientation. In the presence of multiple high-intensity orientations, a single feature can be split into multiple features, each with a different orientation.

[0039] SIFT feature descriptors can be calculated by checking the 16x16 windows around each feature. For 16 4x4 windows in the 16x16 window, gradient amplitude and direction can be calculated and added to the 8 box histograms separated. In some cases, the amplitude representing orientation in the weighted histogram can be based on the proximity to the feature. When filling 8 box histograms for each 4x4 window, 128 values ​​representing the feature can be obtained (16 windows, each window filling 8 histogram blocks, giving 128 total values). These 128 values ​​can be normalized (e.g., by dividing by the total square root) to give a final 128-dimensional vector for each SIFT feature identified in a plurality of SIFT features identified. Therefore, each SIFT feature detected in the image can have a two-dimensional pixel position, scale, orientation and SIFT feature descriptor. However, in other embodiments, feature descriptors of other types can be used.

[0040] Turn again Figure 2 , at 206, method 200 includes generating a plurality of blurred features. Each blurred feature includes a 2D line that passes through a 2D point at which the image feature was detected in an original, unblurred image (e.g., image 100). In addition, each blurred feature includes a feature descriptor that describes the image feature through which the 2D line of the blurred feature passes. In other words, and as will be discussed in more detail below, each image feature detected in the image can be represented by a 2D line that passes through the 2D position of the image feature and a feature descriptor that describes the image feature. The 2D line and feature descriptor together comprise the blurred feature. The blurred representation of the entire image will include a plurality of such blurred features, each blurred feature corresponding to a different image feature detected in the original, unblurred image.

[0041] This is Figure 6A-6C Now refer to Fig. 6A , black dots corresponding to image features detected in image 100 are provided. Specifically, the black dots represent the 2D locations at which image features corresponding to structure 104 and user 106 are detected.

[0042] Fig. 6A Also included is a schematic representation of a single image feature 600. As with image feature 402, image feature 600 includes a feature descriptor 602 (eg, a 128-dimensional vector), a 2D position 604, a scale 606, and an orientation 608.

[0043] Reference now Figure 6BIn order to blur the underlying geometry associated with each image feature, multiple 2D lines are generated, each 2D line passing through a 2D point represented by a black dot. As shown, an example 2D line 610 passes through the black dot corresponding to image feature 600. In some examples, for each of the 2D lines, the direction of the line is randomly selected independently of the directions of the other 2D lines, so that each 2D line has a random 2D direction. In this way, the original image feature is blurred by the multiple 2D lines.

[0044] In this way and now with reference to Figure 6C , the black dot representing the 2D location where the image feature was detected is no longer shown. In this way, it is no longer possible to distinguish the 2D image location from which the feature descriptor was derived, because any algorithm that attempts to infer the appearance of the original image below the 2D line must guess the true 2D location of each underlying image feature along each 2D line. Typically, there will be a large number of potential solutions, mitigating the risk that the two-dimensional location of the original image feature can be recovered. As shown, 2D line 610 is included in the blurred feature 612 corresponding to image feature 600. Although the blurred feature 612 maintains the feature descriptor 602 describing the image feature 600, the 2D position 604, scale 606 and orientation 608 of the image feature are not included in the blurred feature.

[0045] In some examples, the 2D direction of each 2D line associated with the blurred feature can be selected from a predetermined number of different 2D directions. Any appropriate predetermined number of directions can be established and utilized. For example, 100 different directions can be predetermined and stored by the camera device. Each blurred feature line in the blurred feature line can be randomly oriented using one of the 100 different directions. In another example, a random direction can be selected from 256 predetermined directions so that the direction of each feature line can be represented as an 8-bit value. However, in general, any number of potential random directions can be established and selected.

[0046] Return again Figure 2 At 208, method 200 includes transmitting the blurred image representation to a remote device. As will be described in more detail below, the remote device will have a previously acquired 3D map of the physical environment, the 3D map including a plurality of 3D mapping features. By determining the correspondence between the blurred features and the plurality of 3D mapping features, the remote device can estimate the pose of the camera.

[0047] The blurred image representation includes multiple blurred features, including the above-mentioned 2D lines and feature descriptors. It is worth noting that, for example, Figure 6CAs shown in , the blurred image representation does not include the two-dimensional pixel locations of the image features detected in the original image. For example, when the image features are SIFT features, the blurred image representation may not include the two-dimensional locations, scales, or orientations of the SIFT keypoints. This may make it substantially more difficult (if not impossible) for a remote device or intercepting party to recreate the original image from the blurred image representation.

[0048] This is Figure 3 Schematically illustrated in FIG. 3 , a camera device 300 includes a blurred image representation 311, which in turn includes a plurality of blurred features 312. Each blurred feature includes a 2D line 314 and a feature descriptor 316. This information can be stored by the camera device in any suitable manner. As described above, a feature vector will typically include a set of numbers, each corresponding to a different dimension. A SIFT feature descriptor will include 128 such numbers. At the same time, 2D lines can be mathematically represented in various suitable ways, including in the general form of ax+by+c=0. In this way, each 2D line can be represented using three values ​​(e.g., a, b, c from the equation above), which can define a line relative to the image. For example, relative to the original, unblurred image, the center pixel can be established as the origin, and the horizontal and vertical directions are associated with the x-axis and y-axis, where the 2D line is defined relative to the origin.

[0049] Additionally, as will be discussed in more detail below, in some cases it may be desirable to further blur image features prior to transmitting the blurred image representation. Thus, for each of the feature descriptors in the blurred image representation, the camera device may apply a transposition randomly selected from a set of predefined transpositions known to both the camera device and the remote device. When the feature descriptors are multi-dimensional vectors, applying the transposition may include applying any of a set of predefined modifications to one or more of the values ​​defining the vector. This may provide additional defense against a remote device or malicious actor inferring private information from the blurred image representation.

[0050] Communication between the camera device and the remote device may be accomplished in any suitable manner. As an example, the camera device and the remote device may be communicatively coupled using a wired connection, or may be wirelessly connected via Wi-Fi, Bluetooth, or any other suitable wireless communication protocol. Network 320 may take the form of a local area network (LAN), a wide area network (WAN), a wired network, a wireless network, a personal area network, a cellular network, or a combination thereof, and may include the Internet.

[0051] Return again Figure 2At 210, method 200 includes receiving from the remote device a pose of a camera within a physical environment estimated by the remote device. This may be accomplished by determining a correspondence between blur features and a plurality of 3D mapping features maintained by the remote device, as will be discussed in more detail below. Figure 3 , camera device 300 receives estimated pose 318 from remote device 350 via network 320 .

[0052] Now turn to Figure 7 , illustrates an example method 700 for estimating camera pose. Method 700 includes steps performed by a remote device (e.g., remote device 350). As with method 200, method 700 can be performed by any appropriate computer hardware. In different examples, the remote device can take the form of a server, a networked computer, a game console, a mobile communication device, a desktop computer, a laptop computer, a tablet computer, a set-top box (e.g., a cable box, a satellite box), or any other type of appropriate computing device. The remote device can be implemented as follows Fig.12 The computing system 1200 is described.

[0053] At 702, method 700 includes identifying a 3D map of a physical environment, the 3D map including a plurality of 3D map features representing the physical environment. In some examples, the 3D map features may take the form of a reconstructed 3D point cloud representing the environment. As will be discussed in more detail below, in some cases, the 3D map of the environment itself may be blurred. The 3D map may include any appropriate number of points--in some examples, the 3D point cloud may include 20,000, 30,000 or more 3D points representing various locations on a surface within the physical environment.

[0054] Such a 3D map can be generated in any appropriate manner. Typically, a suitable imaging device that captures multiple images or videos of the physical environment from different locations will be used to map the physical environment. These images can be grayscale images, color images, depth images, etc. As described above, image features can be detected in each image. By detecting image features common to more than one of the captured images, the spatial relationship between the images can be determined. When the same image feature is detected in multiple images captured from different locations, the distance between each image feature and the imaging system can be inferred from the observed parallax offset. Additionally or alternatively, depth information from a depth camera can be used to identify the size of the physical environment and the distance between the imaging system and each image feature. In some embodiments, the position of the imaging system can be tracked when it captures multiple images, providing additional information that can be used to stitch multiple images into a 3D representation of the physical environment.

[0055] As described above, each of the 3D points is associated with a feature descriptor extracted from a source image used to construct a 3D map. The feature descriptor may include a multidimensional vector encoding the appearance of a surface around the 3D point. For example, an image block (such as 32x32 pixels) around a given pixel of a 3D point may encode the appearance of the surface around the point into a more compact feature descriptor.

[0056] It is not necessary to generate feature descriptors at every 3D pixel of the 3D point cloud. In some examples, feature descriptors can be selectively generated at distinguishing features of the mapping (such as corners, highly textured areas, etc.). In this way, feature descriptors of blurred features in the blurred image representation can be more easily matched with corresponding 3D mapping features in the 3D mapping. Additionally, in some examples, the initially generated point cloud can be extracted to reduce the density of 3D points included in the cloud. For example, a subset of representative 3D points can be selected in the point cloud, and unselected 3D points within a predetermined distance of the selected 3D points can be discarded. In this way, the resulting extracted 3D point cloud can be loaded faster and more easily processed by a remote device.

[0057] like Figure 7 As used in the context of a 3D map of a physical environment, a 3D map may be "recognized" in any suitable manner. Typically, recognizing a 3D map includes loading the map into a memory of a remote device so that processing (e.g., camera pose estimation) can be completed. The 3D map may be stored on a remote device, retrieved from a remote source, partially or fully dynamically generated, etc.

[0058] Briefly turn to Figure 3 , the remote device 350 includes a logic machine 352 and a storage machine 354, which holds instructions executable by the logic machine to perform various computing tasks. The storage machine 354 includes a 3D map 356 of the physical environment. The 3D map includes a plurality of 3D map features 358. Each 3D map feature includes a 3D point 360 (i.e., the 3D position of the mapped feature within the 3D map) and a 3D point feature descriptor 362. The logic machine 352 and the storage machine 354 can be implemented using any suitable data processing and data storage technology, respectively. Fig.12 Provides additional details about logical machines and storage machines.

[0059] return Figure 7 At 704, method 700 includes receiving a blurred image representation derived from an image of a physical environment. This may be substantially as described above with respect to Figure 2 proceed as described. Figure 3In FIG. 3 , remote device 350 receives blurred image representation 311 from camera device 300 via network 320 . Representation 311 also includes a plurality of blurred features 312 , including 2D lines 314 and feature descriptors 316 .

[0060] Turn again Figure 7 At 706, method 700 includes determining correspondence between blur features in the blurry image representation and 3D mapped features in the 3D map of the physical environment. One example process for determining such correspondence is given herein, but many variations and alternative processes are possible.

[0061] Typically, the remote device identifies the correspondence between the 2D lines in the fuzzy features and the 3D points of the 3D mapped features. This is done by searching the 3D map for the feature descriptors associated with each fuzzy feature and identifying the 3D mapped features in the 3D point cloud with the most similar feature descriptors. In other words, determining the correspondence can include identifying a set of fuzzy features having feature descriptors that match the feature descriptors of the 3D mapped features in the 3D map. As a result, the 2D line of each fuzzy feature in the set corresponds to a 3D point associated with the 3D mapped feature, giving a set of 2D line to 3D point correspondences. The feature descriptor matching step can be implemented using one of many nearest neighbor matching techniques. The L2 distance between the descriptor vectors can be used to calculate pairwise feature similarity.

[0062] It is worth noting that not all determined correspondences are necessarily correct. In fact, some number of incorrect correspondences may be determined. This is because the underlying feature descriptors may be ambiguous and do not always allow for sufficient distinction between feature descriptors of ambiguous features and multiple feature descriptors associated with 3D points in the mapping.

[0063] The image of the physical environment may include items, objects, people, etc., which are not presented when the physical environment is mapped. Therefore, one or more image features that are not represented in the 3D mapping of the physical environment can be detected in the image. Typically, such fuzzy features will not correspond to any 3D points in the mapping. This may be advantageous in some scenarios because any private details depicted in the image will most likely not correspond to any 3D mapped features in the 3D mapping, thereby limiting the amount of information about private details that can be inferred by the remote device. There may also be some number of 3D points in the mapping that do not match any fuzzy features in the fuzzy image representation.

[0064] about Figure 8A-8C The correspondence between the determined blur features and the 3D mapped features is illustrated. Fig. 8A, an image plane 800 is shown, which includes a single 2D line 802. The 2D line 802 is associated with a blur feature that corresponds to a 3D mapped feature in the 3D map of the physical environment.

[0065] The 2D lines associated with the blurred features can be considered as projections of unique 3D planes that originate from the camera projection center and pass through the 2D lines. In other words, each 3D plane represents the back-projection of the 2D line into 3D space in the camera coordinate system. This is in Figure 8B is shown in the figure, Figure 8B The image plane 800 and the 2D line 802 are shown from another angle. As shown, the 2D line 802 has been back-projected into 3D space as a 3D plane 806 that originates from the camera projection center 804.

[0066] Each correspondence between a 2D line and a 3D point in a 3D map provides a constraint on the camera pose solution space. This is because, given the correct camera pose in the 3D map coordinate system, the back-projected 3D plane for the 2D line must pass through the 3D point in the map associated with the 3D map feature corresponding to the fuzzy feature of the 2D line. Therefore, a single correspondence between a 2D line and a 3D point represents a nonlinear constraint on 6 unknown variables, which correspond to the six degrees of freedom camera pose. Depending on the specific algorithm used, 6 such correspondences can be used to calculate the camera pose, and therefore 6 constraints can be used. The 6 constraints can be written in the form of a multivariate polynomial of 6 variables. In one example, when 6 3D plane to 3D point correspondences are known, the minimum solver can calculate a set of 8 candidate camera poses by solving the following algorithm, which outputs a pose solution based on the 3D plane to 3D point correspondences. This can be done in a similar manner to that described in Ramalingam et al., "A minimal solution for registration of 3D points to 3D planes", 2010, except that a Cayley parameterization can be used for the 3D rotation matrix, which leads to a set of polynomial equations with fewer solutions.

[0067] exist Figure 8C In , black dots corresponding to 3D mapped features 808 are shown in the 3D mapped space outside of the image plane 800. When the camera projection center 804 represents the actual camera pose, the 3D plane 806 will pass through both the 2D line 802 and the 3D mapped features 808. Figure 8C, 3D mapped feature 808 has been resolved to position 810 along 2D line 802. Assuming that the camera projection center corresponds to the actual camera pose, 2D position 810 may represent the 2D position of the image feature detected within the original image.

[0068] As described above, not every 2D line in the blurred image representation will necessarily have a corresponding 3D mapping feature in the 3D mapping. Similarly, not every 3D mapping feature will have a corresponding blurred feature. This can be advantageous in some scenarios because any private information (e.g., face, personal object) included in the image that is not presented when the physical environment is mapped will not correspond to a 3D mapping feature. In a specific scenario, the image features detected in the image 100 associated with the structure 104 may relatively have a corresponding 3D mapping feature in the 3D mapping. Therefore, the camera pose relative to the structure can be determined. However, the image features corresponding to the user 106 will not have a corresponding 3D mapping feature because the user may not be in the depicted position when the physical environment is mapped.

[0069] This is Fig.9A and Fig. 9B is shown in the figure. Fig.9A Reappeared Figure 6C The set of 2D lines shown in . Fig.9A In FIG. 1 , two of these lines have been labeled 900A and 900B. Fig. 9B , the correspondence between several of the 2D lines and 3D mapped features in the 3D mapping of the physical environment has been determined. In this way, several of the 2D lines have been resolved to 2D positions where image features were detected in the original image. In this case, all resolved 2D lines give image features detected on structure 104, including 2D line 900B, which has been located at 2D position 902. However, 2D line 900A corresponds to an image feature associated with user 106 and therefore does not have a corresponding 3D mapped feature in the 3D mapping. Therefore, the remote device cannot identify the 2D position of the image feature through which 2D line 900A passes, which means that the remote device cannot infer details about the appearance of user 106.

[0070] Return briefly Figure 7, at 708, method 700 includes: estimating the pose of the camera in the physical environment based on the determined correspondences between the blurred features and the 3D mapped features. Estimating the camera pose can be an iterative random process, in which the correspondences between the 2D lines and the 3D points are identified for multiple candidate camera poses, and these candidate camera poses are evaluated for their consistency with the underlying correspondence data. In an example process, a random sample consensus (RANSAC) method can be used to estimate the pose of the camera from the determined correspondences. In such a case, a minimum or non-minimum solver can be used to calculate multiple candidate camera poses from a subset of the determined correspondences. However, it should be understood that other appropriate methods that do not use the RANSAC framework can be used to estimate the camera pose.

[0071] First, the remote device may identify a subset of correspondences from the entire set of determined 2D line to 3D point correspondences. Fig.10 is schematically shown in Fig.10 The set of all correspondences 1000 identified as described above is included. Correspondences 1000 include a first subset 1002A and a second subset 1002B, which may each have any suitable number of 2D line to 3D point correspondences.

[0072] In some cases, the number of correspondences in the identified subsets may correspond to the minimum number of data points necessary to compute a finite number of solutions using a method known as a minimal solver. For example, when fitting a 2D line to 2D points, a minimum of two unique data points are required to find a unique 2D line (a single solution) that passes exactly through the 2D points. On the other hand, a single data point may be consistent with an infinite number of potential solutions. Thus, for this line fitting toy example, two data points is the minimum number required for a minimal solver. In contrast, a non-minimal solver may use more than two data points.

[0073] Continuing with the 2D line fitting example, multiple subsets of 2D data point pairs can be selected. In the RANSAC framework, these subsets are randomly selected. If a minimal solver is applied to each subset (i.e., a pair of 2D points), a different potential solution can be found from each subset. However, some potential solutions (2D lines) will be more consistent with the input data than other potential solutions. In other words, some potential lines will pass through more 2D points in the data set than other lines. In RANSAC, for each potential solution identified by the minimal solver, the solution (i.e., 2D line) is compared with other 2D data points in the data set to determine which points are consistent with the proposed solution (i.e., "inliers") and which points are inconsistent with the proposed solution (i.e., "outliers"). Any solution with the highest inlier-outlier ratio or with at least a threshold number of total inliers can be accepted as the actual solution - for example, a 2D line accepted as the best fit to a set of 2D points.

[0074] Returning to the camera pose estimation problem, based on the available information, the minimal solver can output a finite number of solutions or potential pose candidates from the determined minimal subset of correspondences. In other words, for each identified subset of correspondences, the remote device can calculate a set of candidate camera poses that satisfy the subset of correspondences. For example, when the subset of correspondences includes 6 2D line to 3D point correspondences, the minimal solver can output 16 candidate camera poses. The 6-point solver can be implemented by solving a system of polynomial equations using a Grobner basis technique. Examples of such techniques are described in "Minimal Solutions for 3D Point Registration to 3D Planes" by Ramalingam et al., 2010, and "A theory of minimal 3D point to 3D plane registration and its generalization" by Ramalingam and Taguchi, 2013. As Fig.10 As shown in , a set of candidate camera poses 1004 are identified from subset 1002A and subset 1002B. The first subset 1002A of correspondences gives two candidate camera poses 1006A and 1006B, while the second subset 1002B gives another two candidate camera poses 1006C and 1006D.

[0075] As an alternative to the 6-point minimum solver, a linear pose estimation method can be used that relies on 12 2D line to 3D point correspondences and outputs only a single solution or a single candidate camera pose. Although this method may involve additional RANSAC iterations compared to methods using a 6-point minimum solver, each iteration can be computationally more efficient, reducing the overall processing time. An example of a suitable linear pose estimation method is described in "Direct 6-DoF Pose Estimation from Point-Plane Correspondences" by Khoshelham, 2015.

[0076] In some cases, when some information about the camera's pose is known, the number of correspondences required by the minimum solver can be reduced. For example, when capturing an image of the physical environment, the camera device can detect a gravity vector relative to its current pose (e.g., via an onboard 3-axis accelerometer). In some cases, this gravity vector can be included in the blurred image representation along with the 2D line and associated feature descriptors. Therefore, two degrees of freedom (corresponding to roll and pitch) of the camera's 3D orientation may already be known, reducing the complexity of estimating the remaining degrees of freedom. For example, when the gravity vector is known, a linear pose estimation method similar to the above method may only require 5 correspondences and output a single solution, or output a candidate camera pose estimated based on the known gravity vector. Similarly, a 4-point minimum solver may only require four correspondences and output two candidate camera poses.

[0077] Once candidate camera poses are computed for each subset of correspondences, the remote device can identify the best overall camera pose from the set of candidate camera poses. In other words, several camera pose hypotheses are identified by the remote device, at least one of which may correspond to the actual pose of the camera.

[0078] exist Fig.10 , pose A 1002A is identified as the best overall camera pose. Typically, the best overall camera pose will be more consistent with the overall set of correspondences than the other candidate camera poses. In other words, for each candidate camera pose, the remote device can determine how many of the determined correspondences will be inliers and how many will be exliers, and can identify camera poses that are consistent with the candidate subset based on this.

[0079] For example, the remote device can assign a RANSAC score to each candidate camera pose by evaluating an objective function that projects the points into a hypothetical image plane within a 3D map of the physical environment, where the projection is based on the candidate camera pose being tested. Given the correspondence between the blurred features and the 3D points, the geometric error (e.g., in pixels) can be measured by calculating the distance between each 2D line and the corresponding 2D point obtained by projecting the 3D points of the 3D mapped features onto the hypothetical image plane. Candidate camera poses with smaller average geometric errors are preferred, and the candidate camera pose with the smallest geometric error can be identified as the best overall camera pose.

[0080] So far, this description focuses on the scene where the camera captures a single image of the physical environment. The camera device derives a fuzzy image representation from the single image, which is transmitted to the remote device and is used to estimate the pose of the camera. However, in alternative embodiments, the camera can capture two or more images of the physical environment, and the fuzzy 3D local map can be derived from the two or more images. Alternatively, for example, in the case of a multi-camera (e.g., stereo camera) device, two or more images can be captured simultaneously by two or more cameras. Therefore, the remote device can estimate the current pose of the camera, which corresponds to the camera pose when the latest image is captured. Additionally or alternatively, the remote device can estimate: the camera pose of each captured image in the captured images, the range of poses depicting the movement of the camera throughout the physical environment when the image is taken, a single pose that is consistent with both images (i.e., in a multi-camera embodiment), etc. As used herein, "3D local mapping" refers to a 3D representation of the local environment of the camera device, which is generated by the camera device based on the images taken by the camera. The 3D local map is distinct from the 3D map of the physical environment maintained by the remote device.

[0081] exist Fig.11A A multi-image scene is illustrated in , which shows two different images 1100 and 1102 of a captured physical environment 102. A number of image features have been identified in each image, whose locations are again represented by black dots. However, it is worth noting that the two images are captured from different perspectives. Therefore, the remote device can first identify 2D feature point correspondences between the two captured images to generate a 3D local map. Such correspondences can be identified by finding image features with matching or highly similar feature descriptors in two or more images, and locating the image features to 3D points via triangulation. In Fig.11A, dashed lines extending from the center of the camera projection and passing through black points corresponding to 2D image features are shown. The dashed lines converge at a 3D point, one of which is labeled 1104, which represents a 3D location corresponding to multiple image features - i.e., a 3D point distributed as a 3D point throughout the 3D local map.

[0082] The 3D local map will be limited to the portion of the physical environment visible in the two or more images, and may therefore be less detailed than the 3D map of the physical environment used by the remote device. Furthermore, the 3D local map may use a different coordinate system than the remote 3D map of the physical environment.

[0083] A 3D local map may be computed from two or more captured images in any suitable manner. Typically, a camera device will detect and match image features that depict common scene points in two or more images. A 3D point cloud reconstruction of the physical environment may be inferred by observing the relative position of each image feature in each captured image, the extent to which each image feature moves as the camera pose changes (e.g., due to parallax), via geometric triangulation, additional information available to the camera device (e.g., depth information), etc.

[0084] Once generated, the 3D local map comprises a 3D point cloud, each of which has an associated feature descriptor computed from image features detected in the images used to construct the 3D local map. Fig. 11B , which shows 3D points triangulated from image 1100 and image 1102. One such 3D point 1104 includes a feature descriptor 1106 and 3D position coordinates 1108, which may be defined relative to the coordinate system of the 3D local map. The feature descriptor 1106 of the 3D point may be the same as a feature descriptor describing one of the 2D image features detected in the two or more images from which the 3D local map is derived.

[0085] Thus, the camera device can derive a blurred 3D local map including a plurality of blurred features. Similar to the blurred features included in the blurred image representation described above, the blurred features in the blurred 3D local map can include: a 3D feature representation that replaces a 3D point in the unblurred 3D local map; and a feature descriptor that describes the appearance of the replaced 3D point.

[0086] The 3D feature representation in the fuzzy feature can take the form of 3D lines, and each 3D line can optionally have a random three-dimensional direction. Fig. 11C , where a 3D line passing through the 3D points of the 3D local map is depicted. Fig.11D, the 3D points are no longer included but are completely replaced by blurred features, one of which is labeled blurred feature 1110. As shown, blurred feature 1110 includes 3D line 1112 and still includes feature descriptors 1106 that describe 2D image features detected in two or more images from which the 3D points replaced by the 3D lines are calculated.

[0087] Alternatively, the 3D feature representation in the blurred feature may take the form of 3D planes passing through the 3D points corresponding to the image features, and each 3D plane may have a random 3D direction.

[0088] Turn again Fig.11A , image 1100 and image 1102 are used to generate a fuzzy 3D local map 1114. This includes a plurality of fuzzy features 1116, which include a 3D feature representation 1118 and a feature descriptor 1120. As described above, the 3D feature representation can be implemented as a 3D line or a 3D plane. It is noteworthy that the fuzzy 3D local map can exclude the original 3D positions of 3D points corresponding to image features detected in two or more images used to construct the 3D local map. However, the fuzzy 3D local map can include any additional information that can support the estimation of one or more camera poses. When two or more images are captured, this can include, for example, an indication of a gravity vector detected by a camera device.

[0089] Once transmitted to the remote device, the fuzzy 3D local map can be used to estimate the camera pose based on the determined correspondence between the 3D feature representation in the fuzzy 3D local map and the 3D points in the 3D map of the physical environment. This typically involves finding a 3D rigid body transformation between the coordinate system of the fuzzy 3D local map and the 3D map of the physical environment. However, the details can be substantially similar to the pose estimation method described above for the fuzzy image representation. For example, in the case where the fuzzy 3D local map includes a 3D plane, then 6 3D plane to 3D point correspondences can be used by a 6-point minimum solver to output 16 potential 3D rigid body transformations. From these transformations, one or more camera poses can be estimated. Alternatively, when the fuzzy 3D local map includes a 3D line, then a set of three 3D line to 3D point correspondences can be used to estimate the 3D rigid body coordinate transformation. This can be described as a generalized absolute camera pose problem. One example approach is described in "A Minimal Solution to the Generalized 3-Point Pose Problem" by Nister and Stewenius, 2003. When the gravity vector detected by the camera is included in the blurred environment representation, then the 3D rigid body transformation can be estimated using only two 3D line-to-3D point correspondences.

[0090] In theory, the fuzzy 3D local map can have an arbitrary scale and does not have to have the same scale as the 3D map of the physical environment used by the remote device. Unless the relative scale factor is known, this will complicate the task of calculating the camera pose. However, these problems can be alleviated when using visual inertial simultaneous positioning and mapping (VI+SLAM) technology or using SLAM with one or more stereo cameras (which reconstruct the world in real dimensions) to generate a 3D local environment map used by the remote device and a 3D map of the physical environment. In such an embodiment, the fuzzy environment representation will have the same scale as the 3D map of the physical environment.

[0091] So far, this description focuses on the case where the 3D mapping of the physical environment includes discrete 3D points distributed throughout the 3D mapping. However, in alternative scenarios, the 3D mapping of the physical environment itself can be blurred. For example, this can be done to prevent anyone who obtains the 3D mapping of the physical environment from recreating the original image used to generate the mapping, as described above. To this end, the 3D points in the 3D mapping of the physical environment can be replaced by 3D lines that all have random directions to blur it. It is worth noting that the 3D lines are selected in a way that makes the 3D lines pass through the 3D points in the original, unblurred version of the 3D mapping, even if the blurred 3D mapping may not include the actual 3D coordinates of the 3D points. When compared with the blurred 3D local mapping derived from one or more images of the physical environment, the RANSAC-based method for determining the 3D rigid body transformation between the local mapping and the original 3D mapping may require a generalized relative pose solver, which outputs 64 candidate solutions when provided with 6 3D line-to-3D line correspondences for the relative 3D rigid body transformation between the coordinate systems of the local mapping and the original 3D mapping. An example approach is described in "Solutions to Minimal Generalized Relative Pose Problems" by Stuvenius et al., 2005. When the gravity vector is known, different minimal solvers provided with four 3D line-to-3D line correspondences can output eight candidate camera poses. Using both confidential queries and confidential mappings may enable untrusted third parties to perform positioning calculations and / or enable positioning calculations to be performed on insecure hardware while guaranteeing the privacy of both mappings and queries (e.g., in scenarios where mappings are exported to a computing device that is spatially close to the physical location of the mapped site, but not necessarily completely secure).

[0092] As described above, when determining the correspondence between blur features and 3D mapping features, any feature representations (e.g., 2D lines, 3D lines, 3D planes) corresponding to items, objects, people, etc. that are not presented in the physical environment during mapping will not have corresponding 3D mapping features in the 3D mapping of the physical environment. Therefore, the positions of image features represented by the blur representation will not be revealed, whether they are in the 2D blur image representation or in the blurry 3D local map. However, the remote device may store any unmatched blur features in the database. When additional blur features are received and additional unmatched blur features are stored, the remote device may eventually be able to determine the correspondence between the various unmatched blur features, potentially allowing the remote device to infer the position or appearance of any object in the image associated with the blur feature. Although this is generally not feasible for objects (e.g., people) that move within the physical environment when the image is captured, the appearance of a stationary object may be revealed in some cases. This problem may be exacerbated when the camera device transmits a blurry 3D local map based on two or more images.

[0093] To address this issue, in some cases, the feature descriptors transmitted along with the blurred features can be modified. As described above, the transmitted feature descriptors can be, for example, 128-dimensional SIFT vectors. Both the camera device and the remote device can maintain a known set of predefined transpositions that can be applied to the feature vectors. Whenever the camera device transmits a blurred image representation to the remote device, the camera device can randomly apply any of these potential transpositions to the included feature descriptors without informing the remote device which transpositions have been selected. In other words, for each feature descriptor in the feature descriptors, the camera device can apply a transposition randomly selected from a known set of predetermined transpositions to the feature descriptors before transmitting the blurred image representation to the remote device.

[0094] Once the remote device receives the blurred image representation, the remote device can apply each predefined transposition in the predefined transposition to each feature descriptor in the received feature descriptor, thereby creating multiple transposed descriptor copies for each feature descriptor. Then, the remote device can search for the correspondence between the transposed descriptor copy and the 3D mapping feature. For image features present in both the captured image and the 3D mapping of the physical environment, at least one transposed copy of the feature descriptor may give a strong feature match-that is, the transposed descriptor copy will at least match the 3D mapping feature using a threshold similarity. Therefore, correspondence can still be determined, which allows the camera pose to be estimated as described above. However, for blurred features that do not have corresponding 3D mapping features in the 3D mapping, any transposed version of the associated feature descriptor is unlikely to result in a strong feature match. In addition, even if, as described above, the unmatched blurred features are stored by the remote device, each stored version of the blurred feature line may also have a different transposition, making it significantly more difficult for the remote device to infer the true feature descriptor and thereby reveal private information.

[0095] In some embodiments, the methods and processes described herein may be linked to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0096] Fig.12 A non-limiting embodiment of a computing system 1200 that can implement one or more of the above methods and processes is schematically shown. The computing system 1200 is shown in a simplified form. Either or both of the camera device 300 and the remote device 350 can be implemented as the computing system 1200. The computing system 1200 can take the form of one or more HMDs, personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices.

[0097] The computing system 1200 includes a logic machine 1202 and a storage machine 1204. The computing system 1200 may optionally include a display subsystem 1206, an input subsystem 1208, a communication subsystem 1210, and / or other subsystems not provided in the computer system. Fig.12 Other components shown in FIG.

[0098] The logic machine 1202 includes one or more physical devices configured to execute instructions. For example, the logic machine can be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions can be implemented to perform tasks, implement data types, transform the state of one or more components, achieve technical effects, or otherwise achieve desired results.

[0099] The logic machine may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processor of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. The individual components of the logic machine may optionally be distributed among two or more separate devices, which may be remotely located and / or configured for coordinated processing. Aspects of the logic machine may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0100] The storage machine 1204 includes one or more physical devices configured to hold instructions executable by a logic machine to implement the methods and processes described herein. When such methods and processes are implemented, the state of the storage machine 1204 may be transformed, for example, to hold different data.

[0101] The storage machine 1204 may include removable and / or built-in devices. The storage machine 1204 may include optical storage (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor storage (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic storage (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.). The storage machine 1204 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices.

[0102] It should be understood that storage 1204 includes one or more physical devices. Alternatively, however, aspects of the instructions described herein may be propagated by a communication medium (eg, electromagnetic signal, optical signal, etc.) that is not retained for a limited duration by a physical device.

[0103] Aspects of the logic machine 1202 and the storage machine 1204 may be integrated together into one or more hardware logic components. Such hardware logic components may include field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0104] The terms "module", "program", and "engine" may be used to describe aspects of the computing system 1200 that are implemented to perform specific functions. In some cases, a module, program, or engine may be instantiated via the logic machine 1202 executing instructions held by the storage machine 1204. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module", "program", and "engine" may cover individuals or groups of executable files, data files, libraries, drivers, scripts, database records, and the like.

[0105] It should be understood that, as used herein, a "service" is an application program that is executable across multiple user sessions. A service may be available for one or more system components, programs, and / or other services. In some embodiments, a service may run on one or more server computing devices.

[0106] When included, display subsystem 1206 can be used to present a visual representation of the data maintained by storage machine 1204. This visual representation can take the form of a graphical user interface (GUI). When the methods and processes described herein change the data stored by the storage machine, and thus convert the state of the storage machine, the state of display subsystem 1206 can also be converted to visually represent the changes in the underlying data. Display subsystem 1206 can include one or more display devices that utilize any type of technology in fact. Such a display device can be combined with logic machine 1202 and / or storage machine 1204 in a shared shell, or such a display device can be a peripheral display device.

[0107] When included, the input subsystem 1208 may include or interface with one or more user input devices, such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the transduction and / or processing of input actions may be handled on-board or off-board. Example NUI components may include microphones for voice and / or sound recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity.

[0108] When included, the communication subsystem 1210 can be configured to communicatively couple the computing system 1200 with one or more other computing devices. The communication subsystem 1210 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem can be configured for communication via a wireless telephone network or a wired or wireless local or wide area network. In some embodiments, the communication subsystem can allow the computing system 1200 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0109] In one example, a method for estimating a camera pose includes: identifying a three-dimensional (3D) map representing a physical environment, the 3D map including a plurality of 3D map features defined at a plurality of 3D points distributed throughout the 3D map; receiving a blurred image representation derived from an original unblurred image of the physical environment captured by a camera, the blurred image representation including a plurality of blurred features, and each blurred feature including: (i) a two-dimensional (2D) line passing through a 2D point in the original unblurred image at which the image feature is detected; and (ii) a feature descriptor associated with the 2D point, the feature descriptor describing the image feature through which the 2D line of the blurred feature passes; determining a correspondence between the blurred features in the blurred image representation and the 3D map features in the 3D map; and estimating a six-degree-of-freedom pose of the camera in the physical environment based on the determined correspondence. In this example or any other example, determining the correspondence includes: identifying a set of blurred features having feature descriptors that match feature descriptors of 3D mapped features in the 3D map, such that a 2D line of each blurred feature in the set corresponds to a 3D point associated with the 3D mapped feature, giving a set of 2D line to 3D point correspondences. In this example or any other example, based on the determined 2D line to 3D point correspondences, the pose of the camera is estimated as follows: a plurality of candidate camera poses are calculated from a subset of the determined 2D line to 3D point correspondences using a random sample consensus (RANSAC) framework and a minimum or non-minimum solver. In this example or any other example, estimating the pose of the camera includes: identifying a subset of the determined 2D line to 3D point correspondences; and identifying one or more candidate camera poses that are consistent with the determined subset of 2D line to 3D point correspondences. In this example or any other example, estimating the pose of the camera also includes: identifying additional subsets of the determined 2D line to 3D point correspondences; for each additional subset of the additional subsets of the 2D line to 3D point correspondences, identifying one or more additional candidate camera poses; and identifying the best overall camera pose from the candidate camera poses. In this example or any other example, each subset of correspondences includes 6 2D line to 3D point correspondences, and each set of candidate camera poses includes 16 candidate camera poses. In this example or any other example, each subset of correspondences includes 6 2D line to 3D point correspondences, each 2D line is back-projected as a 3D plane in the camera coordinate system, and the minimum solver calculates a set of 8 candidate camera poses by solving the following algorithm, and the algorithm outputs a pose solution based on the 3D plane to 3D point correspondences. In this example or any other example, each subset of correspondences includes 12 2D line to 3D point correspondences, and each set of candidate camera poses includes a single candidate camera pose.In this example or any other example, the blurred image representation includes an indication of a gravity vector detected when the original unblurred image was captured, each subset of correspondences includes 5 2D line to 3D point correspondences, and each set of candidate camera poses includes a single candidate camera pose calculated based on the gravity vector. In this example or any other example, the feature descriptors in the blurred image representation are transposed feature descriptors, the transposed feature descriptors have been modified using a transposition randomly selected from a known set of predefined transpositions, and the method also includes: applying each transposition in the known set of predefined transpositions to the transposed feature descriptors to create multiple transposed descriptor copies; searching for correspondences between each transposed descriptor copy and multiple 3D mapped features; and determining the correspondence between the feature descriptor and the 3D mapped feature if the transposed descriptor copy has at least a threshold similarity with the 3D mapped feature.

[0110] In one example, a method for estimating a camera pose includes: capturing an image of a physical environment via a camera; detecting multiple image features at two-dimensional (2D) pixel positions within the image of the physical environment; generating multiple blurred features corresponding to the multiple image features, each blurred feature including (i) a 2D line passing through the 2D pixel position of the corresponding image feature, and (ii) a feature descriptor describing the image feature associated with the 2D point through which the 2D line of the blurred feature passes; transmitting a blurred image representation to a remote device, the blurred image representation including the multiple blurred features, the blurred image representation not including the 2D pixel positions of the multiple image features; and receiving a six-degree-of-freedom pose of the camera within the physical environment from the remote device, the pose estimated by the remote device based on correspondence between the blurred features and multiple three-dimensional (3D) mapping features, the multiple three-dimensional (3D) mapping features being defined at multiple 3D points within a 3D map of the physical environment previously acquired by the remote device. In this example or any other example, each image feature detected in the image has a (2D) position, a scale, a one-dimensional (1D) angular orientation, and a feature descriptor, and the blurred image representation does not include the 2D position, scale, and 1D angular orientation of the image feature. In this example or any other example, the method further includes, for each of the feature descriptors, applying a transposition randomly selected from a predetermined transposition set to the feature descriptor before transmitting the blurred image representation to the remote device. In this example or any other example, each 2D line has a randomly assigned direction.

[0111] In an example, a method for estimating a current camera pose includes: identifying a three-dimensional (3D) map representing a physical environment, the 3D map including a plurality of 3D map features defined at a plurality of 3D points distributed throughout the 3D map; receiving a fuzzy 3D local map derived from two or more images of the physical environment captured by two or more cameras, the fuzzy 3D local map including a plurality of fuzzy features, each fuzzy feature including (i) a 3D feature representation that replaces a 3D point in an unblurred 3D local map, the 3D point having been calculated based on feature matching and triangulation of 2D image features detected in the two or more images, and (ii) a feature descriptor describing one of the 2D image features, the 3D image feature being triangulated from the 2D image feature; determining a correspondence between the fuzzy 3D feature representation in the fuzzy 3D local map and a 3D point feature in the 3D map of the physical environment; and estimating a six-degree-of-freedom pose of the two or more cameras in the physical environment based on the determined fuzzy 3D feature representation to 3D point correspondences. In this example or any other example, the 3D feature representation is a 3D line passing through the 3D points in the original unblurred 3D local map, each 3D line has a random direction, and the determined correspondence includes at least three 3D line to 3D point correspondences. In this example or any other example, the 3D feature representation is a 3D line with a random direction, and the 3D mapping features in the three-dimensional map of the physical environment are fuzzy mapping features, so that each fuzzy mapping feature takes the form of a 3D line with a random direction and passes through the 3D points of the 3D mapping features in the original unblurred version of the 3D map. In this example or any other example, the 3D feature representation is a 3D plane passing through the 3D points in the original unblurred 3D local map, each 3D plane has a random direction, and the determined correspondence includes at least 6 3D planes to 3D point correspondences. In this example or any other example, the fuzzy 3D local map includes an indication of the gravity vector detected when two or more images are captured. In this example or any other example, the fuzzy 3D local map has the same scale as the 3D map of the physical environment.

[0112] It should be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered as restrictive, because many variations are possible. The specific routine or method described herein can represent one or more of any number of processing strategies. Thus, the various actions illustrated and / or described can be performed in the illustrated and / or described order, in other orders, in parallel or omitted. Likewise, the order of the above process can be changed.

[0113] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A method for estimating a camera pose, comprising: identifying a three-dimensional (3D) map representing a physical environment, the 3D map comprising a plurality of 3D map features defined at a plurality of 3D points distributed throughout the 3D map; receiving a blurred image representation derived from an original unblurred image of the physical environment captured by a camera, the blurred image representation comprising a plurality of blurred features, and each blurred feature comprising: (i) a two-dimensional (2D) line passing through a 2D point in the original unblurred image at which the image feature is detected; and (ii) a feature descriptor associated with the 2D point, the feature descriptor describing the image feature through which the 2D line of the blurred feature passes; determining a correspondence between the blur feature in the blurry image representation and the plurality of 3D mapping features in the 3D mapping; and Based on the determined correspondences, a six-degree-of-freedom pose of the camera in the physical environment is estimated.

2. The method of claim 1 , wherein determining the correspondence comprises: A set of blurred features having feature descriptors matching feature descriptors of 3D mapped features in the 3D map is identified such that the 2D line of each blurred feature in the set corresponds to a 3D point associated with the 3D mapped feature, thereby giving a set of 2D line to 3D point correspondences.

3. The method according to claim 2, wherein: Based on the determined 2D line-to-3D point correspondences, the pose of the camera is estimated as follows: a plurality of candidate camera poses are computed from a subset of the determined 2D line-to-3D point correspondences using a random sample consensus (RANSAC) framework and a minimal or non-minimal solver.

4. The method of claim 3, wherein estimating the pose of the camera comprises: identifying a subset of the determined 2D line to 3D point correspondences; as well as One or more candidate camera poses that are consistent with the determined subset of the 2D line to 3D point correspondences are identified.

5. The method of claim 4, wherein estimating the pose of the camera further comprises: Identify an additional subset of the determined 2D line to 3D point correspondences; for For each of the additional subsets of 2D line to 3D point correspondences, identifying one or more additional candidate camera poses; and identifying a best overall camera pose from the candidate camera poses. 6 . The method of claim 3 , wherein each subset of correspondences comprises 6 2D line-to-3D point correspondences and each set of candidate camera poses comprises 16 candidate camera poses.

7. The method of claim 3, wherein each subset of correspondences comprises 6 2D line to 3D point correspondences, each 2D line is back-projected into a 3D plane in the camera coordinate system, and a minimal solver computes a set of 8 candidate camera poses by solving an algorithm that outputs a pose solution based on the 3D plane to 3D point correspondences.

8. The method of claim 3, wherein each subset of correspondences comprises 12 2D line-to-3D point correspondences and each set of candidate camera poses comprises a single candidate camera pose.

9. The method of claim 3, wherein the blurred image representation comprises an indication of a gravity vector detected when the original unblurred image was captured, each subset of correspondences comprises 5 2D line to 3D point correspondences, and each set of candidate camera poses comprises a single candidate camera pose calculated based on the gravity vector.

10. The method of claim 1, wherein the feature descriptors in the blurred image representation are transposed feature descriptors that have been modified using a transposition randomly selected from a known set of predefined transpositions, and the method further comprises: applying each transposition of the known set of predefined transpositions to the transposed feature descriptor to create a plurality of transposed descriptor copies; searching for correspondence between each transposed descriptor copy and the plurality of 3D mapping features; and determining a correspondence between the feature descriptor and the 3D mapped feature if the transposed descriptor copy has at least a threshold similarity to the 3D mapped feature.

11. A method for estimating a camera pose, comprising: Capturing images of the physical environment via a camera; detecting a plurality of image features at two-dimensional (2D) pixel locations within the image of the physical environment; generating a plurality of blur features corresponding to the plurality of image features, each blur feature comprising (i) a 2D line passing through a 2D pixel position of a corresponding image feature, and (ii) a feature descriptor describing the corresponding image feature associated with the 2D pixel position through which the 2D line of the blur feature passes; transmitting a blurred image representation to a remote device, the blurred image representation including the plurality of blurred features, the blurred image representation excluding the 2D pixel locations of the plurality of image features; as well as A six-degree-of-freedom pose of the camera within the physical environment is received from the remote device, the six-degree-of-freedom pose being estimated by the remote device based on correspondences between the blur features and a plurality of three-dimensional 3D mapping features defined at a plurality of 3D points within a 3D map of the physical environment previously acquired by the remote device.

12. The method of claim 11, wherein each image feature detected in the image has a 2D position, a scale, a one-dimensional 1D angular orientation, and a feature descriptor, and the blurred image representation does not include the 2D position, the scale, and the one-dimensional angular orientation of the image feature.

13. The method according to claim 11, further comprising: For each of the feature descriptors, a transposition randomly selected from a set of predetermined transpositions is applied to the feature descriptor prior to transmitting the blurred image representation to the remote device. The method of claim 11 , wherein each 2D line has a randomly assigned direction.

15. A method for estimating a current camera pose, comprising: identifying a three-dimensional (3D) map representing a physical environment, the 3D map comprising a plurality of 3D map features defined at a plurality of 3D points distributed throughout the 3D map; receiving a blurred 3D local map derived from two or more images of the physical environment captured by two or more cameras, the blurred 3D local map comprising a plurality of blurred features, each blurred feature comprising (i) a 3D feature representation, the 3D feature representation being a local 3D point that replaces a local 3D image feature in a corresponding unblurred 3D local map, the unblurred 3D local map comprising a plurality of local 3D image features at a plurality of local 3D points, each local 3D point having been computed based on feature matching and triangulation of a plurality of 2D image features detected in the two or more images, and (ii) a feature descriptor describing one of the 2D image features, the local 3D image feature being triangulated from the one 2D image feature; determining a correspondence between the 3D feature representation in the blurred 3D local map and the plurality of 3D points of the plurality of 3D mapped features in the 3D map of the physical environment; as well as Based on the determined correspondences, a six-degree-of-freedom pose of the two or more cameras in the physical environment is estimated.

16. The method of claim 15, wherein the 3D feature representation is a 3D line passing through the plurality of local 3D points of the plurality of local 3D image features in the unblurred 3D local map, each 3D line having a random direction, and the determined correspondences include at least three 3D line-to-3D point correspondences.

17. A method according to claim 15, wherein the 3D feature representations are 3D lines each having a random direction, and the multiple 3D mapped features in the 3D map of the physical environment are blurred mapped features, so that each blurred mapped feature takes the form of one of the 3D lines, the one 3D line having a random direction and passing through a 3D point of the 3D mapped feature in an unblurred version of the 3D map.

18. The method of claim 15, wherein the 3D feature representation is a 3D plane passing through the plurality of local 3D points in the unblurred 3D local map, each 3D plane having a random direction, and the determined correspondences include at least 6 3D plane to 3D point correspondences.

19. The method of claim 15, wherein the blurred 3D local map includes an indication of a gravity vector detected when the two or more images were captured.

20. The method of claim 15, wherein the blurred 3D local map has the same scale as the 3D map of the physical environment.

Images