Continuous surface and depth estimation

Abstract

Systems, methods, and non-transitory computer-readable media for continuous surface and depth estimation are disclosed. A continuous surface and depth estimation system determines a depth and surface normal of a physical object by using stereo vision limited in a predetermined window.

Description

[0001] Related applications

[0002] This patent application claims the benefit of priority to U.S. Patent Application Serial No. 63 / 190,140, ​​filed May 18, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The implementation of this topic generally relates to computer vision, and more specifically, to continuous surface and depth estimation. Background Technology

[0004] Augmented reality (AR) provides a digitally enhanced experience in which digital content is used to augment a user's real-world environment. For example, virtual content can provide users with data describing their surrounding physical environment, such as presenting information about nearby businesses, providing directions, or displaying weather information. To create the illusion that the virtual content actually exists in the real world, it is displayed to the user based on the distance and orientation of physical objects in their real-world environment. For instance, virtual content can be presented as if it appears superimposed or adjacent to real-world objects related to the virtual content. Attached Figure Description

[0005] In the accompanying drawings (which are not necessarily drawn to scale), the same reference numerals may describe similar parts in different views. The same reference numerals with different letter suffixes may indicate different instances of similar parts. Some embodiments are shown by way of example, not limitation, in the various figures of the accompanying drawings, in which:

[0006] Figure 1 A block diagram of a system for continuous surface and depth estimation according to some example implementations is shown.

[0007] Figure 2 This is a block diagram of a continuous surface and depth estimation system based on some example implementations.

[0008] Figure 3A and Figure 3B This is a flowchart of a method for estimating continuous surfaces and depths according to some example implementations.

[0009] Figure 4 This is a block diagram illustrating components of a machine, according to some example embodiments, capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any or more of the methods discussed herein.

[0010] Figure 5This is a block diagram illustrating components of a machine, according to some example embodiments, capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any or more of the methods discussed herein. Detailed Implementation

[0011] In the following description, various details are set forth for illustrative purposes in order to provide a thorough understanding of some exemplary embodiments. However, it will be apparent to those skilled in the art that the subject matter can be practiced without these specific details or with minor modifications.

[0012] The reference to "one embodiment" or "implementation" in the specification means that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the subject matter. Therefore, the appearance of the phrase "in one embodiment" or "in an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment.

[0013] For illustrative purposes, specific configurations and details have been set forth to provide a thorough understanding of the subject matter. However, it will be apparent to those skilled in the art that embodiments of the described subject matter can be practiced without the specific details presented herein or in various combinations as described herein. Furthermore, well-known features may be omitted or simplified to avoid obscuring the described embodiments. Various examples may be given throughout the specification. These examples are merely descriptions of particular embodiments. The scope or meaning of the claims is not limited to the examples given.

[0014] This article uses the term "augmented reality" (AR) to refer to interactive experiences in real-world environments where physical objects existing in the real world are "enhanced" or strengthened by computer-generated digital content (also known as virtual or synthetic content). AR can also refer to systems that combine the real and virtual worlds, real-time interaction, and three-dimensional registration of virtual and real objects. The user perceives AR systems as virtual content attached to or interconnected with physical objects in the real world.

[0015] Understanding the physical environment is a key element of AR applications. One of the most fundamental ways to gain knowledge about the surrounding physical environment is to estimate the structure of nearby surfaces and objects. Current technologies are computationally expensive and / or require specialized hardware and sensors, such as depth sensors that consume additional power. However, AR devices are preferably designed to be small in size for ease of use by users, so the available computing hardware and sensors may be limited.

[0016] What is disclosed are systems, methods, and non-transitory computer-readable media for continuous surface and depth estimation. The continuous surface and depth estimation system determines the depth and surface normals of a physical object. As previously mentioned, current techniques for determining the depth of an object are computationally expensive and / or require specialized hardware and sensors, such as power-consuming depth sensors. The continuous surface and depth estimation system alleviates these problems by using stereo vision limited to a predetermined window.

[0017] Unlike technologies that rely on depth sensors, stereo vision allows for the extraction of three-dimensional information from digital images. To utilize stereo vision, two optical sensors are displaced relative to each other at known locations and used to capture overlapping images depicting two different views of a real-world environment from two different viewpoints. The relative depth of the captured objects in the images is determined by comparing the relative positions of objects in the two images. For example, the depth of an object can be estimated using triangulation, using a known distance between the two optical sensors, a known viewpoint between the two sensors, and the relative position of the object in the captured images.

[0018] To further reduce computational resource consumption, the continuous surface and depth estimation system restricts the use of stereo vision to a predetermined window within the image captured by the optical sensor. For example, the predetermined window could be a sub-section of the image located at the center of the image captured by the optical sensor. Restricting the use of stereo vision to a predetermined window allows its use even with limited computational resources.

[0019] Continuous surface and depth estimation systems use stereo vision to identify a set of matching features in a pair of corresponding images captured by an optical sensor. Matching features are identifiable points (such as unique regions) of physical objects in a real-world environment, like corners, edges, etc. The continuous surface and depth estimation system identifies features within a predetermined window in one image and then searches for the same features in the corresponding image.

[0020] A continuous surface and depth estimation system determines the depth value for each pair of matching features identified in each corresponding image. For example, a continuous surface and depth estimation system uses the feature's location in the image and the known orientation of the optical sensors (e.g., the distance between optical sensors and the observation point of the optical sensors) to triangulate the feature's depth. The resulting set of depth values ​​is then used to estimate the surface plane indicating the depth of the surface and the surface normal of the physical object. For example, a continuous surface and depth estimation system uses methods such as Random Sample Consensus (RANSAC) to determine the surface plane of the object.

[0021] In some cases, a continuous surface and depth estimation system may not be able to identify a sufficient number of matching features in a pair of corresponding images to determine the surface plane of an object. In this type of case, the continuous surface and depth estimation system can estimate the object's depth based on the available matching features and utilize the surface normals from the previous set of corresponding images to determine the surface plane. If even the number of matching features is insufficient to determine the object's depth (e.g., no matching features are identified), the continuous surface and depth estimation system can use ray projection to determine the surface plane. For example, the continuous surface and depth estimation system can project rays toward a previously known surface plane (such as the last known surface plane) to determine the object's depth.

[0022] In some implementations, the AR device can be configured to render augmentations (i.e., media content) at a relatively high frame rate, while the corresponding sensor data (i.e., image data generated by the optical sensor) is provided at a lower frame rate. To address this issue, the estimated positions of surface planes in each subsequent frame rendered by the system can be forward-predicted to account for unavailable data. As an illustrative example, if the AR device applies a rendering frame rate of 60 Hz, but the optical sensor only provides images at 30 Hz, the estimated positions of surface planes in each subsequent frame rendered by the system can be forward-predicted by forward propagating previously known surface planes (e.g., using a Kalman filter or a double exponential smoothing filter).

[0023] Figure 1 A block diagram of an AR device 100 for continuous surface and depth estimation according to some example embodiments is shown. The AR device 100 provides functionality to enhance a user's real-world environment. For example, the AR device 100 allows a user to observe real-world objects in their physical environment alongside virtual content, thereby enhancing the user's environment. The virtual content can provide the user with data describing their surrounding physical environment, such as data describing nearby businesses, providing directions, displaying weather information, etc.

[0024] Virtual content can be presented to users based on the distance and orientation of physical objects within their real-world environment. For example, virtual content can be presented as if superimposed on the surface of a real-world object. As an example, virtual content describing a recipe can be presented as if superimposed on a kitchen countertop. As another example, virtual content providing directions to a destination can be presented as if superimposed on the surface of the path the user will follow to reach that destination (e.g., a street, the ground).

[0025] In some implementations, AR device 100 may be a mobile device, such as a smartphone or tablet, that presents a real-time image of the user's physical environment along with virtual content. Alternatively, AR device 100 may be a wearable device, such as a helmet or glasses, that allows virtual content to be presented within the user's field of vision, thereby allowing the user to simultaneously observe both the virtual content and the real-world environment.

[0026] As shown in the figure, the AR device 100 includes a left optical sensor 102, a right optical sensor 104, and a display 106. The left optical sensor 102, right optical sensor 104, and display 106 are connected to and configured to communicate with the AR processing system 108 via a communication link 110. The communication link 110 can be physical or wireless. For example, the communication link 110 may include physical wiring or cables connecting the left optical sensor 102, right optical sensor 104, and display 106 to the AR processing system 108. Alternatively, the communication link 110 may be a wireless link facilitated by a wireless communication protocol such as Bluetooth.

[0027] Each of the left optical sensor 102, right optical sensor 104, display 106, and AR system 108 may include one or more devices capable of network communication with other devices. For example, each device may include Figure 5 Some or all of the features, components and peripheral devices of the machine 500 shown in the figure.

[0028] To facilitate communication with other devices, each device includes a communication interface configured to receive communications such as requests and data from another device communicating with it and to pass those communications to appropriate modules or components running on the device. The communication interface also sends communications to another device communicating with it.

[0029] The left optical sensor 102 and the right optical sensor 104 can be any type of sensor capable of capturing image data. For example, the left optical sensor 102 and the right optical sensor 104 can be a camera device configured to capture images and / or video. The images captured by the left optical sensor 102 and the right optical sensor 104 are provided to the AR processing system 108 via the communication link 108.

[0030] To allow for the use of stereoscopic vision, the left optical sensor 102 and the right optical sensor 104 are shifted relative to each other by a known distance to capture overlapping images depicting two different views of the real-world environment from two different viewpoints. The orientation of the optical sensors 102 and 104 within the AR device 100 is calibrated to provide a known image transformation between the two optical sensors 102 and 104. Image transformation is a function that maps the position of a pixel in one image to the corresponding position of the pixel in the corresponding image.

[0031] To enable image transformation to properly map pixel positions between images, optical sensors 102 and 104 are positioned at a predetermined distance from each other and aligned to capture specific observation points. The observation point of each optical sensor 102, 104 indicates the field of view and focus captured by the optical sensors 102, 104. The known distance between optical sensors 102, 104 and the known observation point of each optical sensor 102, 104 can be used to calculate the transformation between images captured by each of the optical sensors 102, 104.

[0032] The display 106 can be any type of display capable of displaying virtual content. For example, the display 106 can be a monitor or screen on which virtual content and an image of the user's physical environment can be displayed simultaneously. Alternatively, the display 106 can be a transparent display that allows the user to view the virtual content displayed on the display 106 as well as real-world objects within the user's field of vision.

[0033] AR processing system 108 is configured to provide AR functionality to enhance the user's real-world environment. For example, AR processing system 108 generates virtual content based on the physical location of surrounding real-world objects and renders this virtual content on display 106 to enhance the user's real-world environment. AR processing system 108 renders the virtual content on the display in a manner that creates the illusion of the virtual content being overlaid on the physical object. For example, AR processing system 108 may generate virtual content based on a defined surface plane that indicates the depth and surface normal of the physical object. Depth indicates the distance between the object and AR device 100, and the surface normal is a vector perpendicular to the surface of the object at a specific point. AR processing system 108 uses the surface plane to generate virtual content and renders the virtual content to create the illusion of the virtual content being overlaid on the surface of the object.

[0034] As mentioned above, current techniques for determining the depth and surface normals of objects are computationally expensive and / or require specialized hardware and sensors, such as power-consuming depth sensors. AR processing system 108 alleviates these problems by using a continuous surface and depth estimation system 112. The continuous surface and depth estimation system 112 uses stereo vision to determine the surface planes of an object within a finite, predetermined window of an image.

[0035] Unlike technologies that rely on depth sensors, stereo vision allows the extraction of three-dimensional information from digital images. For example, two optical sensors 102 and 104 are used to capture a pair of corresponding images depicting two different views of a real-world environment from two different viewpoints. A continuous surface and depth estimation system 112 determines the relative depth of the captured objects in the images by comparing the relative positions of objects in the two images. For example, the depth of the objects can be estimated using triangulation, using a known distance between the two optical sensors 102 and 104, a known viewpoint of the two optical sensors 102 and 104, and the relative positions of the objects in the captured images.

[0036] To further reduce computational resource consumption, the continuous surface and depth estimation system 112 restricts the use of stereo vision to a predetermined window within the images captured by the optical sensors 102 and 104. For example, the predetermined window may be a sub-section of the image located at the center of the images captured by the optical sensors 102 and 104. Therefore, the continuous surface and depth estimation system 112 focuses its function on objects captured within the predetermined window, and may not recognize features or objects not present within the predetermined window.

[0037] To achieve this, the continuous surface and depth estimation system 112 identifies a set of matching features in a pair of corresponding images captured by optical sensors 102 and 104. Matching features are identifiable points of physical objects in the real-world environment, such as corners, edges, etc. The continuous surface and depth estimation system 112 first identifies features within a predetermined window in one of the images (e.g., the image captured by the left optical sensor 102), and then searches for the same features in the corresponding image (e.g., the image captured by the right optical sensor 104).

[0038] The continuous surface and depth estimation system 112 determines the depth value for each pair of matching features identified in each of the corresponding images. For example, the continuous surface and depth estimation system 112 uses the location of the features in the images and the known orientations of the optical sensors 102, 104 (e.g., the distance between them and the viewpoint) to triangulate the depth of the features. The resulting set of depth values ​​is then used to estimate the surface plane indicating the depth of the surface and the surface normal of the physical object. For example, the continuous surface and depth estimation system 112 uses methods such as Random Sample Consensus (RANSAC) to determine the surface plane of the object.

[0039] In some cases, the continuous surface and depth estimation system 112 may not be able to identify a sufficient number of matching features in a pair of corresponding images to determine the surface plane of an object. In this type of case, the continuous surface and depth estimation system 112 can estimate the depth of the object based on the available matching features and use the surface normals of the previous set of corresponding images to determine the surface plane. If the number of matching features is even insufficient to determine the depth of the object, the continuous surface and depth estimation system 112 can use ray projection to determine the surface plane. For example, the continuous surface and depth estimation system 112 can project rays toward a previously known surface plane of the object (e.g., the last known surface plane) to determine the depth of the object.

[0040] The continuous surface and depth estimation system 112 provides data defining the determined surface planes to the AR processing system 108. The AR processing system 108 can then use the determined surface planes to generate and render virtual content that appears to be superimposed on the surface of an object.

[0041] Figure 2 This is a block diagram of a continuous surface and depth estimation system 112 according to some example embodiments. To avoid obscuring the subject matter of the invention with unnecessary detail, Figure 2 Various functional components (e.g., modules) not closely related to conveying an understanding of the inventive subject matter have been omitted. However, those skilled in the art will readily recognize that the continuous surface and depth estimation system 112 can support various additional functional components to facilitate additional functions not specifically described herein. Furthermore, Figure 2 The various functional modules described can reside on a single computing device or can be distributed across several computing devices in various arrangements (e.g., arrangements used in cloud-based architectures).

[0042] As shown, the continuous surface and depth estimation system 112 includes an image access unit 202, a feature recognition unit 204, a depth determination unit 206, a surface plane determination unit 208, and an output unit 210.

[0043] Image access unit 202 accesses corresponding images captured by left optical sensor 102 and right optical sensor 104. Corresponding images are a pair of two images captured by the two optical sensors 102 and 104 at approximately the same time. The corresponding images captured by optical sensors 102 and 104 can be used to determine the depth of objects using stereo vision. To utilize stereo vision, the two optical sensors 102 and 104 are horizontally shifted relative to each other and are used to capture images depicting two different views of the real-world environment from two different viewpoints. The corresponding images accessed by image access unit 202 depict the physical environment of AR device 100 from the viewpoints of the optical sensors 102 and 104 that captured the corresponding images. For example, the image captured by left optical sensor 102 depicts the physical environment from the viewpoint of left optical sensor 102, and the image captured by right optical sensor 104 depicts the physical environment from the viewpoint of left optical sensor 104.

[0044] Image access unit 202 can access corresponding images from left optical sensor 102 and right optical sensor 104 directly or via AR processing system 108. For example, left optical sensor 102 and right optical sensor 104 can provide images directly to image access unit 202 using communication link 110. As another example, left optical sensor 102 and right optical sensor 104 can provide images to AR processing system 108, which then provides the images to image access unit 202, or store the images in memory from which image access unit 202 can access them. Image access unit 202 provides the accessed corresponding images to other components of continuous surface and depth estimation system 112.

[0045] The feature recognition unit 204 identifies matching features in a pair of corresponding images. A feature is an identifiable part or component of an image. Features that can be identified in an image include edges, corners, points of interest, spheres, ridges, etc. Matching features in two images are features identified in each image that are determined to be the same. For example, a matching feature could be an edge or corner identified in each image as a part of the same real-world environment depicted in the image.

[0046] The feature recognition component 204 uses any of the various known feature detection techniques or algorithms used for image processing to recognize features. For example, the feature recognition component 204 may use a technique that identifies features based on the contrast of nearby pixels and / or patterns in the image.

[0047] The feature recognition unit 204 first identifies multiple features within a predetermined window in one of the images. The predetermined window can be a sub-part of the image. For example, the predetermined window can be the portion located at the center of the images captured by optical sensors 102 and 104. The feature recognition unit 204 identifies features within the predetermined window in one of the images (e.g., the image captured by the left optical sensor 102) and then attempts to find matching features in the corresponding image (e.g., the image captured by the right optical sensor 104). For example, the feature recognition unit 204 can utilize a transformation function to identify matching features in the corresponding image. The feature recognition unit 204 can compare the features identified in each of the two images to determine whether the features match. For example, the feature recognition unit 204 can analyze the pixels depicting each feature based on color, pattern, etc., to determine whether the two features match.

[0048] The feature recognition unit 204 provides data identifying the set of matching features to other components of the continuous surface and depth estimation system 112. For example, the data may include location data describing the relative positions of features in each image. This may include coordinates describing the pixel positions depicting the matching features in each image, data identifying the optical sensor that captured each feature (e.g., the left optical sensor 102 or the right optical sensor 104), the time the feature was captured, etc.

[0049] Depth determination component 206 determines a depth value indicating the depth of a matching feature identified by feature recognition component 204. Depth determination component 206 uses stereo vision to determine the depth value. For example, it uses triangulation to determine the depth value of each matching feature by comparing the relative positions of the matching features in two images. For example, it uses the position of the matching feature in each image and the known orientation of the optical sensors 102, 104 (such as the distance between optical sensors 102, 104, and the observation point of optical sensors 102, 104) to determine the depth value of the matching feature. The depth value indicates the depth of the physical features of an object relative to the AR device 100.

[0050] The depth determination component 206 provides the obtained set of depth values ​​to the surface plane determination component 208. The surface plane determination component 208 uses this set of depth values ​​to estimate the surface plane that indicates the depth and surface normal of the physical object's surface. For example, the surface plane determination component 208 uses methods such as Random Sample Consensus (RANSAC) to determine the surface plane of the object.

[0051] In some cases, there may not be a sufficient number of matching features for the surface plane determination component 208 to determine the surface plane of a physical object. For example, in a pair of corresponding images, the feature recognition component 204 may have already identified a limited number of matching features (e.g., fewer than three). In this type of case, the surface plane determination component 208 can estimate the depth of the object based on the available matching features (e.g., based on the depth values ​​determined by the depth determination component 206 from the pair of corresponding images) and utilize the surface normals in the previous set of corresponding images to determine the surface plane. If the number of matching features is even insufficient to determine the depth of the object, the surface plane determination component 208 can use ray projection to determine the surface plane. For example, the surface plane determination component 208 can project rays toward a previously known surface plane (such as the last known surface plane) to determine the depth of the object.

[0052] The output unit 210 provides data defining the determined surface plane to the AR processing system 108. The AR processing system 108 can then use the determined surface plane to generate and render virtual content that appears to be superimposed on the surface of an object.

[0053] Figure 3A and Figure 3B This is a flowchart of a method for continuous surface and depth estimation according to some example embodiments. Methods 300, 330 can be implemented in computer-readable instructions for execution by one or more computer processors, such that the operation of methods 300, 330 can be performed partially or entirely by the continuous surface and depth estimation system 112; therefore, methods 300, 330 are described below by way of example with reference to the continuous surface and depth estimation system 112. However, it should be understood that at least some of the operations of methods 300, 330 can be deployed on various other hardware and / or software configurations, and methods 300, 330 are not intended to be limited to the continuous surface and depth estimation system 112.

[0054] Figure 3A A method 300 for continuous surface and depth estimation is illustrated. At operation 302, a feature recognition unit 204 detects a set of matching features within a predetermined window of the corresponding image. Features are identifiable portions or parts of an image. Examples of features that can be identified in an image include edges, corners, points of interest, spheres, ridges, etc. Matching features in two images are features identified as identical in each image. For example, a matching feature could be an edge or corner identified in each image as depicting the same portion of a real-world environment portrayed in the image.

[0055] The feature recognition component 204 uses any of the various known feature detection techniques or algorithms used in image processing to identify features. For example, the feature recognition component 204 may use techniques that identify features based on the contrast of nearby pixels and / or patterns in the image.

[0056] The feature recognition unit 204 first identifies multiple features within a predetermined window in one of the images. The predetermined window can be a sub-part of the image. For example, the predetermined window can be the portion located at the center of the images captured by optical sensors 102 and 104. The feature recognition unit 204 identifies features within the predetermined window in one of the images (e.g., the image captured by the left optical sensor 102) and then attempts to find matching features in the corresponding image (e.g., the image captured by the right optical sensor 104). For example, the feature recognition unit 204 can utilize a transformation function to identify matching features in the corresponding image. The feature recognition unit 204 can compare the features identified in each of the two images to determine whether the features match. For example, the feature recognition unit 204 can analyze the pixels depicting each feature based on color, pattern, etc., to determine whether the two features match.

[0057] At operation 304, depth determination component 206 determines the depth value for each pair of matching features. Depth determination component 206 uses stereo vision to determine the depth values. For example, by comparing the relative positions of matching features in two images, triangulation is used to determine the depth value of each matching feature. For example, the depth value of the matching features is determined using the position of the matching features in each image and the known orientations of optical sensors 102, 104 (such as the distance between optical sensors 102, 104, and the observation points of optical sensors 102, 104). The depth value indicates the depth of the physical features of the object from the AR device 100.

[0058] At operation 306, the surface plane determination component 208 determines the surface plane based on this set of depth values. The surface plane determination component 208 uses this set of depth values ​​to estimate the surface plane that indicates the depth and surface normal of the surface of the physical object. For example, the surface plane determination component 208 uses methods such as Random Sample Consensus (RANSAC) to determine the surface plane of the object.

[0059] In some cases, there may not be a sufficient number of matching features for the surface plane determination component 208 to determine the surface plane of an object. In this type of case, the surface plane determination component 208 can estimate the depth of the object based on the depth values ​​determined for the available matching features and use the surface normals of the previous set of corresponding images to determine the surface plane. If the number of matching features is insufficient to determine the depth of the object, the surface plane determination component 208 can use ray projection to determine the surface plane. For example, the surface plane determination component 208 can project a ray toward the last known surface plane to determine the depth of the object.

[0060] At operation 308, the AR processing system 108 renders virtual content based on a surface plane. The AR processing system 108 is configured to provide AR functionality to enhance the user's real-world environment. For example, the AR processing system 108 generates and renders virtual content on display 106 based on the physical location of surrounding real-world objects to enhance the user's real-world environment. The AR processing system 108 renders virtual content on the display in a manner that creates the illusion of virtual content overlaid on physical objects. For example, the AR processing system 108 may generate virtual content based on a defined surface plane that indicates the depth and surface normal of the surface of a physical object. The depth indicates the distance of the object from the AR device 100, and the surface normal is a vector perpendicular to the surface of the object at a specific point. The AR processing system 108 uses the surface plane to generate and render virtual content to create the illusion of virtual content overlaid on the surface of an object.

[0061] Figure 3B A method 330 for continuous surface and depth estimation is shown when there is not a sufficient number of matching features for the surface plane determination component 208 to determine the surface plane of an object.

[0062] At operation 332, the feature recognition unit 204 detects a set of matching features in a predetermined window of the corresponding image.

[0063] At operation 334, the depth determination component 206 determines the depth value for each pair of matching features.

[0064] At operation 336, the surface plane determination component 208 determines that the number of matching features in the set of matching features is lower than a threshold number of features used to determine the surface plane. For example, the surface plane determination component 208 may require at least three matching features to accurately determine the surface plane of the object.

[0065] At operation 338, the surface plane determination component 208 determines the surface plane based on the set of depth values ​​and the surface normal of the previous surface plane. The previous surface plane was determined based on a set of matching features identified in a previously captured set of corresponding images.

[0066] Software Architecture

[0067] Figure 4 This is a block diagram illustrating example software architecture 406, which can be used in conjunction with various hardware architectures described herein. Figure 4 This is a non-limiting example of software architecture 406, and it should be understood that many other architectures can be implemented to facilitate the functionality described herein. Software architecture 406 can be implemented in hardware such as... Figure 5 The system executes on a machine 500, which, among other things, includes a processor 504, a memory 514, and (input / output) I / O components 518, etc. A representative hardware layer 452 is shown and can represent, for example... Figure 5 The machine 500 in the document. A representative hardware layer 452 includes a processing unit 454 with associated executable instructions 404. The executable instructions 404 represent executable instructions of the software architecture 406, including implementations of the methods, components, etc., described herein. Hardware layer 452 also includes a memory and / or storage module 456, which also has the executable instructions 404. Hardware layer 452 may also include other hardware 458.

[0068] exist Figure 4 In the example architecture, software architecture 406 can be conceptualized as a stack of layers, each providing a specific function. For example, software architecture 406 may include layers such as operating system 402, libraries 420, framework / middleware 418, applications 416, and presentation layer 414. Operationally, applications 416 and / or other components within a layer can activate application programming interface (API) calls 408 and receive responses such as messages 412 in response to API calls 408 via the software stack. The layers shown are representational in nature, and not all software architectures have all layers. For example, some mobile operating systems or dedicated operating systems may not provide framework / middleware 418, while other operating systems may provide such a layer. Other software architectures may include additional or different layers.

[0069] Operating system 402 can manage hardware resources and provide public services. Operating system 402 may include, for example, a kernel 422, services 424, and drivers 426. Kernel 422 can act as an abstraction layer between the hardware layer and other software layers. For example, kernel 422 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 424 can provide other public services to other software layers. Drivers 426 are responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers 426 may include display drivers, camera drivers, etc. Drives, flash drives, serial communication drives (e.g., Universal Serial Bus (USB) drives), Drivers, audio drivers, power management drivers, etc.

[0070] Library 420 provides common infrastructure used by application 416 and / or other components and / or layers. Library 420 provides functionality that allows other software components to perform tasks more easily than by directly interfaceing with the functions of the underlying operating system 402 (e.g., kernel 422, services 424, and / or drivers 426). Library 420 may include system libraries 444 (e.g., the C standard library), which provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, library 420 may include API libraries 446 such as media libraries (e.g., libraries supporting the rendering and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., the OpenGL framework for rendering 2D and 3D graphics content on a display), database libraries (e.g., SQLite providing various relational database functionalities), web libraries (e.g., WebKit providing web browsing functionality), etc. Library 420 may also include various other libraries 448 to provide many other APIs to application 416 and other software components / modules.

[0071] The framework / middleware 418 (sometimes also called middleware) provides a higher level of common infrastructure that can be used by applications 416 and / or other software components / modules. For example, the framework / middleware 418 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. The framework / middleware 418 can provide a wide range of other APIs that can be used by applications 416 and / or other software components / modules, some of which may be specific to a particular operating system 402 or platform.

[0072] Application 416 includes built-in applications 438 and / or third-party applications 440. Examples of representative built-in applications 438 may include, but are not limited to: contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 440 may include those used by entities other than the platform-specific vendor using Android. TM or iOS TM Applications developed using a Software Development Kit (SDK) can be used on platforms such as iOS. TM ANDROID TM , Mobile software running on the phone's mobile operating system or other mobile operating systems. Third-party application 440 may invoke API calls 408 provided by the mobile operating system (e.g., operating system 402) to facilitate the functions described herein.

[0073] Application 416 may use built-in operating system functions (e.g., kernel 422, service 424, and / or driver 426), libraries 420, and frameworks / middleware 418 to create a UI for interacting with the system's user. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 414. In these systems, the application / component "logic" may be separated from the user-interacting aspects of the application / component.

[0074] Figure 5 This is a block diagram illustrating components of a machine 500 according to some example embodiments, which is capable of reading instructions 404 from a machine-readable medium (e.g., a machine-readable storage medium) and executing any or more of the methods discussed herein. Specifically, Figure 5 A graphical representation of a machine 500 in the form of an example computer system is shown, within which instructions 510 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause the machine 500 to perform any or more of the methods discussed herein. Thus, the instructions 510 can be used to implement the modules or components described herein. The instructions 510 transform a general, unprogrammed machine 500 into a specific machine 500 programmed to perform the described and illustrated functions in the described manner. In alternative embodiments, the machine 500 operates as a standalone device or can be coupled to other machines (e.g., networked). In a networked deployment, the machine 500 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 500 may include, but is not limited to, server computers, client computers, PCs, tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart home appliances), other smart devices, web appliances, network routers, network switches, network bridges, or any machine 500 capable of sequentially or otherwise executing instructions 510 specifying actions to be taken by machine 500. Furthermore, although only a single machine 500 is shown, the term "machine" should also be considered to include a collection of machines that individually or jointly execute instructions 510 to perform any or more of the methods discussed herein.

[0075] Machine 500 may include processor 504, memory / storage device 506, and I / O components 518 that can be configured to communicate with each other, for example, via bus 502. Memory / storage device 506 may include memory 514, such as main memory or other storage device, and storage cells 516, which processor 504 can access, for example, via bus 502. Storage cells 516 and memory 514 store instructions 510 that implement any one or more of the methods or functions described herein. Instructions 510 may also reside wholly or partially in memory 514, storage cells 516, at least one processor in processor 504 (e.g., the processor's cache memory), or any suitable combination thereof during execution by machine 500. Thus, memory 514, storage cells 516, and the memory of processor 504 are examples of machine-readable media.

[0076] I / O component 518 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O component 518 included in a particular machine 500 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It will be understood that I / O component 518 may include... Figure 5 Many other components are not shown. The I / O components 518 are grouped by function only for the purpose of simplifying the following discussion, and this grouping is by no means limiting. In various example embodiments, the I / O components 518 may include output components 526 and input components 528. Output components 526 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input components 528 may include alphanumeric input components (e.g., keyboards, touchscreens; photo-optical keyboards, or other alphanumeric input components configured to receive alphanumeric input), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens that provide position and / or force for touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0077] In other example implementations, I / O component 518 may include biometric component 530, motion component 534, environmental component 536, or positioning component 538, as well as various other components. For example, biometric component 530 may include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 534 may include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Environmental component 536 may include, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), a sound sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of hazardous gases to ensure safety or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 538 may include a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer for detecting air pressure from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0078] Various technologies can be used to implement communication. I / O component 518 may include communication component 540, which is operable to couple machine 500 to network 532 or device 520 via coupling 524 and coupling 522, respectively. For example, communication component 540 may include network interface component or other suitable means to interface with network 532. In other examples, communication component 540 may include wired communication component, wireless communication component, cellular communication component, near field communication (NFC) component, etc. Components (e.g.) (low energy consumption) Components and other communication components that provide communication via other means. Device 520 can be any peripheral device from another machine or various peripheral devices (e.g., a peripheral device coupled via USB).

[0079] Furthermore, the communication component 540 can detect identifiers or may include components operable to detect identifiers. For example, the communication component 540 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). Additionally, various information can be obtained via the communication component 540, such as location via Internet Protocol (IP) geolocation, etc. Location can be obtained through signal triangulation or by detecting NFC beacon signals that indicate a specific location.

[0080] Glossary

[0081] In this context, "carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions 510 executed by machine 500, and includes digital or analog communication signals or other intangible media to facilitate the communication of these instructions 510. Instructions 510 can be sent or received via network 532 using a transmission medium and utilizing any of a number of well-known transmission protocols.

[0082] In this context, "client device" refers to any machine 500 that interfaces with communication network 532 to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, PDAs, smartphones, tablet computers, ultrabooks, netbooks, laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, STBs, or any other communication device that a user can use to access network 532.

[0083] In this context, "communication network" refers to one or more parts of network 532, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), LAN, wireless LAN (WLAN), WAN, wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a POTS (Plain Old-Style Telephone Service) network, a cellular telephone network, a wireless network, etc. A network, other types of networks, or a combination of two or more such networks. For example, network 532 or a portion thereof may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, the coupling may implement any of a variety of data transmission technologies, such as Single Carrier Radio Transmission (1xRTT), Evolved Data Optimization (EVDO), General Packet Radio Service (GPRS), Enhanced Data Rate Evolution of GSM (EDGE), the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other standards defined by various standards-setting organizations, other telematics protocols, or other data transmission technologies.

[0084] In this context, "machine-readable medium" means a component, device, or other tangible medium capable of temporarily or permanently storing instructions 510 and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)) and / or any suitable combination thereof. The term "machine-readable medium" should be considered to include a single medium or multiple media capable of storing instructions 510 (e.g., a centralized or distributed database or associated cache memory and server). The term "machine-readable medium" should also be considered to include any medium or combination of media capable of storing instructions 510 (e.g., code) executable by machine 500, such that when executed by one or more computer processors 504 of machine 500, the instructions 510 cause machine 500 to perform any or more of the methods described herein. Accordingly, "machine-readable medium" means a single storage device or apparatus, and a "cloud-based" storage system or storage network comprising multiple storage devices or apparatuses. The term "machine-readable medium" does not include the signal itself.

[0085] In this context, a “component” refers to a device, physical entity, or logic having boundaries defined by functional or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularity for specific processing or control functions. A component can be combined with other components via its interface to perform machine processing. A component can be an encapsulated functional hardware unit designed for use with other components and can be part of a program that typically performs a specific function in a related function. A component can constitute a software component (e.g., code implemented on a machine-readable medium) or a hardware component. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various example implementations, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups 504) of a computer system can be configured by software (e.g., application 416 or application portions) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuitry or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuitry temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor 504 or another programmable processor 504. Once configured by such software, the hardware component becomes uniquely tailored to perform the configured function and is no longer a specific machine 500 (or a specific component of machine 500) of the general-purpose processor 504. It will be understood that decisions to mechanically implement hardware components in dedicated and permanently configured circuitry or in temporarily configured (e.g., software-configured) circuitry may be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate or perform certain operations described herein. Considering the implementation where hardware components are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component at any given time. For example, when the hardware components include a general-purpose processor 504 that can be configured as a dedicated processor via software, the general-purpose processor 504 can be configured as different dedicated processors (e.g., including different hardware components) at different times. Therefore, the software configures a particular processor or multiple processors 504 to constitute a specific hardware component at one time and different hardware components at different times. The hardware components can provide information to and receive information from other hardware components.Therefore, the described hardware components can be considered communicatively coupled. In the presence of multiple hardware components, communication can be achieved through signal transmission between two or more hardware components (e.g., via appropriate circuitry and bus 502). In embodiments where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from that memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Other hardware components can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., information collection). The various operations of the example methods described herein can be performed at least in part by one or more computer processors 504 that are temporarily or permanently configured (e.g., via software) to perform the relevant operations. Whether temporarily or permanently configured, such processors 504 can constitute components of a processor implementation that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more computer processors 504. Similarly, the methods described herein can be implemented at least in part by processors, with a particular processor 504 or processors 504 being an example of hardware. For example, at least some operations of the methods can be performed by one or more computer processors 504 or processor-implemented components. Furthermore, one or more computer processors 504 can also operate to support the execution of related operations in a "cloud computing" environment or as a "Software as a Service" (SaaS) operation. For example, at least some operations can be performed by a group of computers (as an example of a machine 500 including processors 504), and these operations can be accessed via a network 532 (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of certain operations can be distributed among processors 504, residing not only within a single machine 500 but also deployed across multiple machines 500. In some example implementations, processors 504 or processor-implemented components can reside in a single geographic location (e.g., within a home environment, an office environment, or a server cluster). In other example implementations, the processor 504 or the processor-implemented components may be distributed across multiple geographical locations.

[0086] In this context, "processor" refers to any circuit or virtual circuit (physical circuitry simulated by logic executed on the actual processor 504) that manipulates data values ​​according to control signals (e.g., "commands", "opcodes", "machine codes", etc.) and generates corresponding output signals used to operate machine 500. Processor 504 can be, for example, a central processing unit (CPU), a simplified instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), or any combination thereof. Processor 504 can also be a multi-core processor having two or more independent processors 504 (sometimes referred to as "cores") capable of executing instructions 510 simultaneously.

Claims

1. A method for estimating continuous surfaces and depths, comprising: Detect a set of matching features in a predetermined window for a first image and a second image, the set of matching features including a first feature depicted in the predetermined window of the first image and a second feature depicted in the predetermined window of the second image, the first image being captured by a first optical sensor and the second image being captured by a second optical sensor, the set of matching features corresponding to physical features of a physical object; Determine the depth value for each pair of matching features, and generate a set of depth values, each depth value indicating the depth of the corresponding physical feature of the physical object; A surface plane is determined based on the set of depth values, the surface plane indicating the depth and surface normal of the surface of the physical object; as well as Virtual content is rendered based on the surface plane.

2. The method according to claim 1, wherein, The predetermined window corresponds to a sub-part of the first image and the second image.

3. The method according to claim 2, wherein, The sub-part is the central portion of the first image and the second image.

4. The method according to claim 1, wherein, Determining the depth value includes: Determine the position of a certain feature of the physical object in the first image and the position of the feature of the physical object in the second image; and The depth value is triangulated based on the position of the feature in the first image, the position of the feature in the second image, the known orientation of the first optical sensor, and the known orientation of the second optical sensor.

5. The method according to claim 1, wherein, Determining the surface plane based on the set of depth values ​​includes: Determine that the number of matching features in the set of matching features is lower than the threshold number of matching features used to determine the surface plane; and The surface plane is determined based on the set of depth values ​​and the surface normal of a previous surface plane, the previous surface plane being determined based on a set of matching features from a pair of images captured by the first and second optical sensors before the first and second images were captured.

6. The method according to claim 1, wherein, Determining the surface plane based on the set of depth values ​​includes: Determine that the number of matching features in the set of matching features is lower than the threshold number of matching features used to determine the depth of the surface of the physical object; and The surface plane of the object is determined by projecting light rays toward a previously known surface plane of the object.

7. The method according to claim 1, wherein, Rendering the virtual content based on the surface plane includes: The virtual content is overlaid when the surface of the physical object is presented on the client device.

8. A system comprising: One or more computer processors; as well as One or more computer-readable media storing instructions that, when executed by the one or more computer processors, cause the system to perform operations, the operations including: Detect a set of matching features in a predetermined window for a first image and a second image, the set of matching features including a first feature depicted in the predetermined window of the first image and a second feature depicted in the predetermined window of the second image, the first image being captured by a first optical sensor and the second image being captured by a second optical sensor, the set of matching features corresponding to physical features of a physical object; Determine the depth value for each pair of matching features, and generate a set of depth values, each depth value indicating the depth of the corresponding physical feature of the physical object; A surface plane is determined based on the set of depth values, the surface plane indicating the depth and surface normal of the physical object's surface; and Virtual content is rendered based on the surface plane.

9. The system according to claim 8, wherein, The predetermined window corresponds to a sub-part of the first image and the second image.

10. The system according to claim 9, wherein, The sub-part is the central portion of the first image and the second image.

11. The system according to claim 8, wherein, Determining the depth value includes: Determine the position of a certain feature of the physical object in the first image and the position of the feature of the physical object in the second image; and The depth value is triangulated based on the position of the feature in the first image, the position of the feature in the second image, the known orientation of the first optical sensor, and the known orientation of the second optical sensor.

12. The system according to claim 8, wherein, Determining the surface plane based on the set of depth values ​​includes: Determine that the number of matching features in the set of matching features is lower than the threshold number of matching features used to determine the surface plane; and The surface plane is determined based on the set of depth values ​​and the surface normal of a previous surface plane, the previous surface plane being determined based on a set of matching features from a pair of images captured by the first and second optical sensors before the first and second images were captured.

13. The system according to claim 8, wherein, Determining the surface plane based on the set of depth values ​​includes: Determine that the number of matching features in the set of matching features is lower than the threshold number of matching features used to determine the depth of the surface of the physical object; and The surface plane of the object is determined by projecting light rays toward a previously known surface plane of the object.

14. The system according to claim 8, wherein, Rendering the virtual content based on the surface plane includes: The virtual content is overlaid when the surface of the physical object is presented on the client device.

15. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of one or more computing devices, cause the one or more computing devices to perform operations, the operations including: Detect a set of matching features in a predetermined window for a first image and a second image, the set of matching features including a first feature depicted in the predetermined window of the first image and a second feature depicted in the predetermined window of the second image, the first image being captured by a first optical sensor and the second image being captured by a second optical sensor, the set of matching features corresponding to physical features of a physical object; Determine the depth value for each pair of matching features, and generate a set of depth values, each depth value indicating the depth of the corresponding physical feature of the physical object; A surface plane is determined based on the set of depth values, the surface plane indicating the depth and surface normal of the surface of the physical object; as well as Virtual content is rendered based on the surface plane.

16. The non-transitory machine-readable storage medium according to claim 15, wherein, The predetermined window corresponds to a sub-part of the first image and the second image.

17. The non-transitory machine-readable storage medium according to claim 16, wherein, The sub-part is the central portion of the first image and the second image.

18. The non-transitory machine-readable storage medium according to claim 15, wherein, Determining the depth value includes: Determine the position of a certain feature of the physical object in the first image and the position of the feature of the physical object in the second image; and The depth value is triangulated based on the position of the feature in the first image, the position of the feature in the second image, the known orientation of the first optical sensor, and the known orientation of the second optical sensor.

19. The non-transitory machine-readable storage medium according to claim 15, wherein, Determining the surface plane based on the set of depth values ​​includes: Determine that the number of matching features in the set of matching features is lower than the threshold number of matching features used to determine the surface plane; and The surface plane is determined based on the set of depth values ​​and the surface normal of a previous surface plane, the previous surface plane being determined based on a set of matching features from a pair of images captured by the first and second optical sensors before the first and second images were captured.

20. The non-transitory machine-readable storage medium according to claim 15, wherein, Determining the surface plane based on the set of depth values ​​includes: Determine that the number of matching features in the set of matching features is lower than the threshold number of matching features used to determine the depth of the surface of the physical object; and The surface plane of the object is determined by projecting light rays toward a previously known surface plane of the object.

Images