Systems and methods for anchoring virtual objects to physical locations

Abstract

The present disclosure provides systems and methods that allow a first user to anchor a virtual object to a physical location and allow a second user to view the virtual object at the physical location. In particular, aspects of the present disclosure are directed to workflows, application programming interfaces ("APIs"), and computing system infrastructure that enable a first computing device to generate a self-organizing map of its physical environment (e.g., based on sensor data collected by the first device) and position and anchor virtual objects at physical locations within such a map. Data describing the map and virtual objects can be stored in a cloud database and then used by a second device or for the second device to render a virtual object at the location of the anchor.

Description

[0001] Related Applications

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 668,205, filed on May 7, 2018. The entire contents of U.S. Provisional Patent Application No. 62 / 668,205 are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to augmented reality or virtual reality. More specifically, the present disclosure relates to systems and methods that enable users to anchor virtual objects to physical locations. Background Art

[0004] Virtual reality may refer to a computer-generated environment that is experienced through sense and perception simulation. Augmented reality may refer to a real-time view of a physical real-world environment that has been "augmented" with computer-added sensory information, which may be referred to as virtual objects. For example, computer-added virtual objects may include information designed to be experienced through one or more sensory modalities, such as, for example, vision, hearing, touch, somatosensory, and / or smell.

[0005] While some virtual or augmented reality systems do provide a degree of shared experience across multiple users, these systems typically rely on a top-down model of virtual environment generation and deployment. This top-down model does not allow users to generate and anchor virtual objects within the physical world. Such models also do not allow user-anchored virtual objects to be shared at scale, making them inefficiently rendered across multiple attached devices. Summary of the invention

[0006] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0007] One example aspect is directed to one or more server computing devices configured to perform operations. The operations include receiving, from a first computing device, data describing an environment surrounding the first computing device and data describing a pose of an anchor within the environment surrounding the first computing device, wherein a virtual object is associated with the anchor. The operations include storing the data describing the pose of the anchor in a data storage device. The operations include Figure 1A -D receiving a request for a pose of the anchor from the second computing device. The operation includes providing data describing the pose of the anchor relative to the pose of the second computing device to the second computing device.

[0008] Another example aspect is directed to a computing device configured to perform operations. The operations include collecting sensor data describing an environment of the computing device. The operations include receiving a request from a user to generate an anchor for a virtual object within the environment of the computing device. The operations include providing data describing the environment of the computing device and data describing a pose of the anchor for the virtual object within the environment of the computing device to a server computing system via an application programming interface.

[0009] Another example aspect is directed to a computing device configured to perform operations. The operations include receiving data describing a pose of an anchor of a virtual object within an environment of the computing device from a server computing system via an application programming interface. The operations include rendering the virtual object within the environment of the computing device on a display of the computing device according to the pose of the anchor.

[0010] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0011] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] A detailed discussion of embodiments for those of ordinary skill in the art is set forth in the specification with reference to the accompanying drawings, in which:

[0013] Figure 1A-Figure 1D An example computing system configuration is depicted according to an example embodiment of the present disclosure.

[0014] Figure 2 Depicted is a block diagram of an example workflow for an augmented reality system, according to an example embodiment of the present disclosure.

[0015] Figure 3 A block diagram of an example computing system is depicted according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] In general, the present disclosure is directed to systems and methods that allow a first user to anchor a virtual object to a physical location and allow a second user to view the virtual object at the physical location. In particular, aspects of the present disclosure are directed to workflows, application programming interfaces ("APIs"), and computing system infrastructure that enable a first computing device to generate a self-organizing map of its physical environment (e.g., based on sensor data collected by the first device) and to locate and anchor virtual objects at physical locations within such a map. Data describing the map and the virtual object can be stored in a cloud database and then used by a second device or for the second device to render the virtual object at the location of the anchor. For example, the rendering of the second device can be based on a second self-organizing map of the second device's physical environment (e.g., generated based on sensor data collected by the second device).

[0017] More particularly, a central aspect of the present disclosure is the placement and use of "anchors." An anchor may generally refer to a connection defined between the physical world and the virtual world. The use of anchors allows for the placement and positioning of virtual content within the physical world (e.g., by an individual user), and subsequently allows for the rendering of the anchored virtual content on the physical world at the anchor location.

[0018] Thus, an anchor typically corresponds to or includes a pose (e.g., location and orientation) in the real world (e.g., the corner of a particular table). For example, a pose can be expressed in a coordinate frame such as a Cartesian coordinate frame. The pose can be surfaced via a software development kit ("SDK") using an API in a manner that allows the user to attach a virtual object to the pose. For example, the system can allow the user to place a virtual object on the corner of a table.

[0019] After the anchor has been placed, it can be stored in a cloud system and can then be served to the attached device (e.g., via an API). Thus, when the attached device seeks to enjoy an AR experience at a particular table, the virtual object will reappear in the exact same location. This remains true even if the application associated with the device that placed the anchor has been closed.

[0020] In order for this recognition to work, the computing device may be enabled to describe / understand what the scene around the anchor looks like. Thus, in some embodiments, each device may collect sensor data (e.g., images captured by (one or more) cameras) and may memorize or otherwise determine the geometry and appearance of the surrounding scene in a manner that enables the device to be positioned relative to the anchor.

[0021] Thus, the systems and methods described herein support augmented reality ("AR") and / or virtual reality ("VR") multi-user "shared" experiences (e.g., where sessions occur substantially simultaneously) and / or "persistent" experiences (e.g., where multiple sessions of one or more users occur simultaneously and / or at different times), where virtual objects anchored to the physical world by a first user are made visible to one or more second users present at the location of the anchor. In particular, a cloud infrastructure including cross-platform APIs and SDKs enables seamless shared experiences across multiple different devices and users.

[0022] In some embodiments, anchors generated by a user may be described in a data structure called a Frame of Interest (FOI). The FOI may describe the pose (e.g., position and orientation) at which a virtual object has been anchored. In some embodiments, the location at which one or more virtual objects have been anchored may be referred to as an AR or VR "hotspot."

[0023] In some embodiments, the anchor can be positioned relative to one or more visual features (e.g., descriptors and detectors) in the environment. In some embodiments, the computing device operated by the user who places the anchor (e.g., its application) may need to view the anchor from any perspective from which they want the anchor to be recognized later. The device and / or application can collect visual information actively (e.g., during a mapping step, for example, where the user is provided with instructions to walk around the AR content location) or passively (e.g., during regular application use). In some embodiments, the anchor may also include or be associated with an image or video clip to help the user identify the location where the anchor is placed (e.g., this may be particularly useful when sharing with other users who will need to find the AR anchor). This additional image collection and storage may also help develop better visual features.

[0024] In some embodiments, the FOI describes the pose of the anchor, but not necessarily the virtual object itself. Instead, for example, the model of the virtual object can be simply referenced and accessed from some other location (e.g., a cloud database, a collection of fixed assets contained in the application, and / or some other channel). Thus, the AR artifacts can be positioned relative to the anchor. AR artifacts can be stored in a first-party artifact index, or they can be managed by a third-party application or application backend. Anchors can support all of these use cases and do not necessarily require the use of a first-party index.

[0025] The pose (e.g., position and orientation) of an anchor may be defined based on a map of or relative to a portion of the physical world. The map may be a previously generated global map, or according to one aspect of the present disclosure, the map may be a local map generated based on sensor data collected by a device seeking to establish an anchor.

[0026] In some implementations, a map of a portion of the physical world may be stored and described in a data structure called an Area Description File (ADF). The ADF may include a map of the environment that describes the appearance and geometry of the scene in a way that allows an AR rendering algorithm to recognize the scene and / or position the device relative to the scene.

[0027] In some embodiments, the ADFs are not publicly exposed. ADFs may be merged with other ADFs and may be enhanced with additional data over time. For example, merging two or more ADFs may include combining the data provided by the two or more ADFs into a single file with a single shared coordinate frame. In some cases, if the environments described by the ADFs overlap at least partially, they may be merged.

[0028] According to one aspect, the present disclosure provides an API and system infrastructure that supports persistent placement of virtual content in the physical world without the need for pre-mapped space. In particular, the present disclosure provides a computing infrastructure that enables cloud-based storage and service of information useful for anchoring and rendering virtual objects, such as, for example, environment maps and virtual object descriptions.

[0029] As an example, a server computing system may provide a Visual Positioning Service (VPS), which may optionally be provided as a service in the "cloud". The VPS may generate, receive, store, use and / or serve FOIs describing anchor poses of virtual objects and ADFs describing underlying mapping data. The VPS may manage, merge, optimize and / or compress FOIs and / or ADFs. For example, an ADF may be shared and / or indexed by its corresponding FOI.

[0030] Thus, in some embodiments, a computing system (e.g., a VPS provided by a server computing system) can perform management of the FOI. For example, in some embodiments, a FOI that should be displayed in the current session can be requested through an API. The user computing device can then load the necessary mapping data for a given FOI from a local disk and / or through the VPS (e.g., using tiles or server-side queries), and can render virtual objects within the scene according to the FOI.

[0031] According to another aspect, the computing system (e.g., a VPS provided by a server computing system) can implement persistence of the FOI. For example, in some embodiments, if the FOI should be persisted, the underlying mapping system can collect necessary sensor data (e.g., raw and / or processed video data for identification and / or positioning) during FOI creation. The persistent FOI can be stored in a FOI store. Under the API, multiple FOIs can reference the same mapping data, and the mapping data of multiple FOIs can be merged into a single representation.

[0032] In some embodiments, the computing system can provide persistence to support the VPS. For example, in some embodiments, FOIs and / or ADFs can be uploaded to the VPS backend for indexing and processing. Overlapping ADFs can be merged and jointly perfected. An improved and / or compressed version of the mapping data can be downloaded back to the "original" device, allowing the device to free up some memory in its memory. Once the FOIs are persisted to the cloud, they can also be shared with other devices for coexistence experiences or other experiences. ADFs that overlap with the global VPS scaffolding can also be registered to the global scaffolding.

[0033] In some embodiments, in order for multiple sessions (e.g., occurring at the same or different times) to consistently show virtual objects at the same physical location, all involved electronic devices (e.g., user devices such as smartphones, head mounted displays, etc.) must understand their pose (e.g., location and orientation) relative to the environment. Therefore, the devices largely establish or otherwise understand a common reference frame and know their pose within it. Once this common reference frame is established and real-time pose estimation is enabled, all devices can render relevant content in a consistent manner.

[0034] In some cases, pose co-registration between the above multiple devices can be achieved by using external tracking devices (such as those used in motion capture studios). However, for more widespread mobile AR / VR systems adopted by ordinary end users, the use of complex external tracking devices is generally not practical.

[0035] In view of this deficiency, the present disclosure provides systems and methods for establishing a common reference frame using visual, inertial, and / or other sensory information captured by a device's (one or more) cameras, (one or more) inertial measurement units ("IMUs"), and / or other sensors. In an example workflow, a "hosting device" uses the sensor information to build a model of the environment and build a map of the environment (e.g., which can take the form of an ADF), which is sent to a server computing system (e.g., a VPS of a server system) in raw or pre-processed form. At the server system, the information is potentially further processed before being used in conjunction with data from a "receiving device" to identify visual elements in the environment.

[0036] In some embodiments, these processing and identification steps may be performed by a server system (e.g., in which case the receiving device may send sensor data to the server for identification). Alternatively, the processing and identification steps may be performed by the receiving device (e.g., in which case the server system sends sensory information from the hosting device to the receiving device for processing). In further embodiments, in addition to paths involving a server system, a direct point-to-point connection between devices may alternatively be possible, where two devices directly exchange visual information to establish a common reference frame. Here, data processing may be performed by the hosting device and / or the receiving device.

[0037] Once a common reference frame is established through matching of sensor data, the anchor pose (and hence the positioning of the content) is resolvable for both devices and allows consistent rendering of virtual content across both devices. Thus, Cloud Anchors enables persistent content placement and discovery at a global scale.

[0038] In particular, in some embodiments where the system is cross-platform, the cloud anchor service API can interface with low-level APIs for different platforms (such as Google's ARCore or Apple's ARKit). The system can capture and / or send motion and visual data via an API at a fixed or variable frequency. The information may involve raw and / or pre-processed sensor information from a camera, IMU, and / or other sensor. The raw and / or pre-processed and / or compressed information is then sent to a server backend where the data is further processed and stored. In some embodiments, when the receiving device wants to resolve an anchor post, it sends the visual data to a server or hosting device, which then compares the information with a visual map created by the hosting device. Once a match is found, the calculated post of the receiving device can be returned. Together with the anchor pose initially created, the desired content can be rendered.

[0039] Thus, in some embodiments, a receiving computing device (e.g., a user computing device such as a smart phone, head mounted display, or other rendering device) can perform FOI localization and / or can work with a server or other hosting device to perform FOI localization. For example, in some embodiments, as long as an application on the device accesses the FOI coordinate system, the underlying mapping data will be retained in memory and used for localization of the FOI on the device. By switching to the most appropriate ADF internally, the localization algorithm can most accurately track the FOI closest to the user. FOIs that are not actively tracked may drift relative to the camera pose.

[0040] For FOIs persisted to the cloud, location queries can also be handled by the VPS backend. An application can send a visual query to the VPS that queries all nearby ADFs in an area. Once an ADF is identified and a device can be located within it, the FOI (and its transformation) is available to the application for content rendering. For example, a server computing system can transform all FOIs to the service start (e.g., initial) frame of the device to simplify developer workflow. In order to accurately track the FOI, the system may decide to download mapping data (e.g., tiles) from the server to the device.

[0041] In some embodiments, the anchors established may have a limited lifespan. For example, in some embodiments, each item of content placed at an AR hotspot may have a limited lifespan, depending on, for example, the richness of the captured sensor data and / or other characteristics. After the content expires, it may be automatically deleted from the system. To make the content last longer, the user may capture more sensor data / viewpoints / time of day / etc. Having an automatic expiration date has good implications for user expectations and backend infrastructure.

[0042] Content can be discovered in many different ways. In one example, content discovery can occur through explicit content sharing. For example, in some embodiments, a user can share a content URL or other indication of the content location, which contains information about where to find the experience. Once in the correct physical location, the VPS server system can recognize the ADF and parse the pose of the content relative to the camera for rendering. Beacon devices are a potential option for performing content sharing. For example, a beacon device can inform all users in a specific space of relevant content.

[0043] In another example, content discovery can occur through visual recognition. For example, in some embodiments, content can be placed publicly; anchored to a stand and / or to its own hotspot ADF. The user can discover content by using the VPS to identify nearby places where objects are placed (e.g., a compass-like user interface can provide guidance to the experience). For ADFs identified by the VPS, the corresponding FOI gesture can be sent to the device. In turn, the content can be parsed using the available FOI.

[0044] Thus, in one example, starting from an approximate location (e.g., GPS), the device can be cloud-positioned for nearby ADFs, and embedded anchors and their poses can be returned in device coordinates. The AR index can then use the set of identified anchors and their locations to retrieve visible content. Applications can use anchor metadata for filtering (e.g., limiting to anchors created or renderable by consuming applications). If the user is not actively using AR, the location can also be used to alert the user to the presence of nearby AR content.

[0045] In one example use, two users, each with a device, look at a scene. They can be at the same physical scene at the same time or at different times. The first user places an object at the detected plane and presses the "Publish" button in the application. A background process (e.g., with a progress notification) is started, which sends the recorded scene appearance and FOI data to the VPS system. Once the progress bar reaches 100%, the content becomes visible to the second user in the same physical space without explicit mapping or sharing steps. In some embodiments, controls can be provided that enable the first user to limit the visibility of the object to a specific subset of users (e.g., the second user). Alternatively, the user can allow visibility of the object to any and all users.

[0046] Example applications that may utilize Cloud Anchors include entertainment applications (e.g., location-based or multi-player (particularly asynchronous or large-scale) games, Tilt Brush art installations, etc.); social expression applications (e.g., persistent decorations (e.g., AR gifts or decorations), graffiti walls, bulletin boards, etc.); information annotation applications (e.g., attaching a schedule to a conference room, instructions for an Airbnb rental, etc.); and / or other types of applications.

[0047] Thus, in some embodiments, the present disclosure provides or utilizes the following technical components: a browsable library of assets for placing source AR indexes; plane detection and user interface for object placement by a first user; on-device and / or cloud FOI management; data logging and uploading (e.g., in the form of compressed sensor streams and / or features, such as, for example, as mp4 video plus IMU data); VPS map processing (e.g., low-latency bundle processing); VPS locator indexing (e.g., fingerprinting and inserting into a dynamic index by sequence); querying the VPS (e.g., dynamic and static indexes) to retrieve parsable FOIs for matching ADFs; retrieving those FOIs that are locatable by a micro ADF or stand; and providing a pose for content and allowing retrieval of FOIs. The system described herein provides the first AR SDK that allows cross-platform sharing and persistence. Sharing can be performed as an automated (cloud backend) service.

[0048] Benefits of using cloud-enabled services (compared to purely on-device services, for example) include:

[0049] Public and asynchronous sharing: Cloud-enabled anchors can be shared broadly without the need for point-to-point connections or application-specific infrastructure.

[0050] Global-scale anchor storage and discovery: Cloud solutions can more efficiently manage global-scale AR content creation, discovery, and retrieval.

[0051] Improved anchor recall and accuracy: Cloud processing allows for the use of more compute- and data-intensive algorithms that leverage usage data to improve performance. Engineers can also innovate faster on cloud-enabled algorithms and deploy to end users without waiting for app updates.

[0052] Cross-platform: Implement once and distribute to different mobile devices with a small device footprint.

[0053] Universal AR Viewing: With a shared index and standardized content format, a single AR viewing app can render content from multiple sources.

[0054] Referring now to the drawings, example embodiments of the present disclosure will be discussed in further detail.

[0055] Example Compute System Configuration

[0056] Figure 1A-Figure 1D Depicted are example computing system configurations according to example embodiments of the present disclosure. In particular, Figure 1AAn example system is depicted in which anchor information and visual data for scene recognition are sent to a server for relay and processing. The "receiving device" then sends sensor information to the server, and the server performs a match on the data from the receiving device with the data of the "hosting device". For example, performing the match may include executing various visual feature matching algorithms designed for localization and / or coordinate alignment. When the match is successful, the anchor and device pose are returned from the server to the "receiving device".

[0057] Figure 1B Another example system is depicted in which anchor information and visual data for scene recognition are sent to a server for relay and processing before being passed to a second device. Recognition of the pose of the "receiving device" is computed on the receiving device where the server sends sensor information from the "hosting device" to the receiving device. Thus, the same as in the example in which visual matching is performed at the server device. Figure 1A In comparison, Figure 1B In the embodiment of the present invention, visual matching is performed at the receiving device.

[0058] Figure 1C Another example system is depicted where anchor information and visual data for scene recognition are passed directly between devices without involving a server. The “receiving device” sends its visual data to the “hosting device” which returns the relative pose of the two devices and the pose of the anchor. Figure 1C In , visual matching is performed at the hosting device.

[0059] Figure 1D Another example system is depicted in which anchor information and visual data for scene recognition are passed directly between devices without involving a server. A “hosting device” sends its visual data and anchor poses to a “receiving device” which performs recognition on its own. Figure 1D In , visual matching is performed at the hosting device.

[0060] Example Workflow

[0061] Figure 2 A block diagram of an example workflow for an augmented reality system is depicted. The illustrated workflow may be implemented at least in part (e.g., in addition to the illustrated devices 200 and 250) by a visual processing system implemented by a server computing system. The illustrated workflow is provided as an example only. Other workflows may also be used.

[0062] First, Alice launches the app on her Android device 200. Next, the ARCore 204 of the device 200 starts running motion tracking.

[0063] Next, Alice places and shares the AR content. Calls ARCore 204 to create an anchor. Calls anchor hosting SDK 202 to share the anchor. VPS SDK 202 calls createAnchor() to VPS server 226 to create a cloud anchor (a). VPS SDK 202 extracts sensor information and creates a dataset (b). VPS SDK 202 uploads the dataset to receiving server 210 (c).

[0064] The dataset is stored in encrypted short-term storage 212 (d). Upload server 216 sends a notification to hotspot processor 214 (e). Hotspot processor 214 receives dataset_uploaded notification and begins optimizing the ADF from storage 212 and saves the resulting map at server storage 218 for serving (f).

[0065] The hotspot processor 214 updates the location information associated with the anchor at the FOI storage 222 (g). The hotspot processor 214 sends a dataset_processed notification, and the upload server 216 responds to the dataset_processed notification by cleaning the dataset from the short-term storage 212 (h).

[0066] Next, Alice shares anchor_id(i) with Bob's iOS device 250. This step may also occur earlier (e.g., after (a)).

[0067] Device 250 launches the app on his iOS device. ARKit 254 motion tracking starts from the local ARKit coordinate frame.

[0068] Device 250 retrieves the AR content shared by Alice. Specifically, Anchor (VPS) SDK 252 calls resolveAnchorPose() with anchor_id, camera_features, sos_t_imu, and imu_t_camera (j). VPS server 226 checks access control 224 (k). Assuming device 250 can access the content, VPS server 226 obtains adf_t_anchor from the anchor and calls Localize() to cloud positioning service 220, which uses the Tile from (f) for positioning. VPS server 226 returns camera_t_anchor to Anchor (VPS) SDK 252 (l). VPS SDK 252 calls ARKit 254 to create an ARAnchor for ARKit tracking.

[0069] Example computing system

[0070] Figure 3 A block diagram of an example computing system 100 is depicted in accordance with an example embodiment of the present disclosure. System 100 includes a user computing device 102, a user computing device 152, and a server computing system 130 communicatively coupled via a network 180. For example, communications between user computing device 102, user computing device 152, and server computing system 130 may be accomplished through the use of an application programming interface.

[0071] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0072] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or a plurality of processors operably connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store instructions 118 and data 116 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0073] The user computing device 102 may include a software development kit (SDK) 120. The SDK may include computer-executable instructions that, when executed by the processor(s) 112, enable the user computing device 102 to share and receive anchor poses for three-dimensional objects. The user computing device 102 may also include an augmented reality application or engine 121. For example, the augmented reality application or engine 121 may include computer-executable instructions that, when executed by the processor(s) 112, enable the user computing device 102 to render augmented reality content (e.g., virtual objects) on a display of the user computing device 102.

[0074] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0075] The user computing device 102 may also include one or more sensors 123, such as a camera, accelerometer, gyroscope, radar, lidar, and / or multiple types of sensors. The sensors 123 may collect information describing the surrounding environment of the user computing device 102 (e.g., visual information).

[0076] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or multiple processors operably connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 134 may store instructions 138 and data 136 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0077] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0078] The server computing system 130 may include a vision processing system 140. The vision processing system 140 may include computer executable instructions that, when executed, cause the server computing system to receive, store, index, and provide pose and environment information associated with a virtual anchor. For example, the vision processing system 140 may implement, at least in part (e.g., in addition to the devices 200 and 250 shown), Figure 2 Workflow shown.

[0079] The visual processing system 140 may communicate with one or more databases, such as: a database 141 storing one or more area description files (ADFs) or other types of environment maps; a database 142 storing one or more frames of interest (FOIs) or other types of anchor pose data; and / or a database 143 storing data that can be used to render one or more virtual objects. The database may be a single database or a distributed database. The virtual objects may be two-dimensional, three-dimensional, fixed-position / shape, dynamic mesh objects, and / or other types of virtual objects.

[0080] Network 180 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications through network 180 can be carried via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0081] Figure 3 An example computing system that can be used to implement the present disclosure is shown. Other computing systems can also be used.

[0082] Additional disclosure

[0083] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and divisions of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system, or can be distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0084] A computing device may refer to any type of computing device, including, for example, a smart phone, a laptop computer, a desktop computer, a server computing device, an embedded computing device, a game console, an intelligent device, etc. A computing device typically includes one or more processors and a memory. The one or more processors may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be a processor or a plurality of processors operably connected. The memory may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory may store instructions and data executed by the processor to enable the computing device to perform operations. The functional aspects of the computing device may be implemented in software, hardware, firmware, or a combination thereof.

[0085] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of illustration, not limitation of the present disclosure. Those skilled in the art, after obtaining an understanding of the foregoing, can easily make changes, variations, and equivalents to such embodiments. Therefore, the present disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter as would be apparent to one of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce another embodiment. Therefore, the present disclosure is intended to cover such changes, variations, and equivalents.

Claims

1. A server computing system comprising one or more server computing devices configured to perform operations, the operations include: receiving, from a first computing device, data describing an environment surrounding the first computing device and data describing a pose of an anchor placed within the environment surrounding the first computing device by a user of the first computing device, wherein a virtual object is associated with the anchor, and wherein the anchor allows placement and positioning of the virtual object within a physical world; storing data describing the pose of the anchor in a data storage device, wherein storing the data describing the pose of the anchor in a data storage device comprises: generating and storing a frame of interest for the anchor, and wherein the frame of interest references a model of a virtual object; receiving a request for a pose of the anchor from a second computing device; and Data describing a pose of the anchor is provided to the second computing device, the data describing the pose of the anchor including a frame of interest of the anchor.

2. The server computing system according to claim 1, in, The one or more server computing devices receive data from and provide data to the first computing device and the second computing device via one or more predefined application programming interfaces.

3. The server computing system according to claim 1, in, The one or more server computing devices receive data and provide the data to respective software development kits implemented by the first computing device and the second computing device.

4. The server computing system according to claim 1, in: The data describing the surrounding environment of the first computing device received by the server computing system from the first computing device includes data describing one or more visual features of the surrounding environment of the first computing device; as well as Providing the second computing device with data describing the pose of the anchor includes providing the second computing device with data describing the pose of the anchor relative to the one or more visual features of the surrounding environment.

5. The server computing system according to claim 1, in: The data describing the surrounding environment of the first computing device received by the server computing system from the first computing device includes data describing one or more first visual features of the surrounding environment of the first computing device; The operations further include: receiving, from the second computing device, data describing one or more second visual characteristics of a second surrounding environment of the second computing device; and matching the one or more first visual features with the one or more second visual features to determine a second pose of the second computing device relative to a surrounding environment of the first computing device; and Providing the second computing device with data describing the pose of the anchor includes providing the second computing device with data describing the pose of the anchor relative to a second pose of the second computing device.

6. The server computing system according to claim 5, in, Providing data describing a pose of the anchor relative to a second pose of the second computing device to the second computing device includes: positioning the second computing device relative to the surrounding environment of the first computing device; and Data describing a pose of the anchor relative to the positional location of the second computing device is provided to the second computing device.

7. The server computing system according to claim 1, in, The interest frame also references an access location at which data enabling rendering of the virtual object can be accessed.

8. The server computing system according to claim 1, in, Storing data describing the pose of the anchor in a data storage device includes indexing the anchor based on the location of the anchor.

9. The server computing system according to claim 1, in, The operations further include generating and storing a region description file based on data describing a surrounding environment of the first computing device.

10. The server computing system according to claim 9, in, The operations further include merging the region description file with one or more other region description files describing overlapping environments.

11. The server computing system according to claim 1, in, The operations further include: receiving an approximate location associated with the second computing device; and A plurality of anchors located proximally to the second computing device is identified.

12. The server computing system according to claim 1, in, The operations further include deleting the anchor after expiration of a predefined time period, wherein the predefined time period is based on characteristics of the sensor data of the surrounding environment.

13. The server computing system according to claim 1, in, The operations further include: maintaining an access control list for the anchor; and Based on the access control list, before providing data describing the pose of the anchor to the second computing device, it is ensured that the second computing device has permission to access the anchor.

14. A first computing device, include: one or more processors; as well as one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the first computing device to perform operations comprising: collecting sensor data describing an environment surrounding the first computing device; receiving a request from a user to generate an anchor for a virtual object within the surrounding environment of the first computing device, wherein the anchor allows placement and positioning of the virtual object within the physical world; and Data describing the surrounding environment of the first computing device and data describing the posture of an anchor for a virtual object within the surrounding environment of the first computing device are provided to a server computing system via an application programming interface, wherein the data describing the posture of the anchor includes an interest frame of the anchor, and wherein the interest frame references a model of the virtual object.

15. The first computing device according to claim 14, in, The first computing device performs the operations by implementing a software development kit included in an application.

16. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a second computing device, cause the one or more processors to perform operations, the operations include: receiving, from the server computing system via the application programming interface, data describing a pose of an anchor of a virtual object within a surrounding environment of a second computing device, wherein the anchor allows placement and positioning of the virtual object within the physical world, and wherein the data describing the pose of the anchor includes a frame of interest of the anchor, and wherein the frame of interest references a model of the virtual object; and A virtual object within a surrounding environment of the second computing device is rendered on a display of the second computing device based on the pose of the anchor and based on the model of the virtual object.

17. One or more non-transitory computer readable media according to claim 16, in, The operations further include: collecting sensor data describing an environment surrounding the second computing device; receiving, from the server computing system via the application programming interface, a region description file describing a surrounding environment of the second computing device; and The second computing device is positioned relative to the pose of the anchor based on the sensor data and the region description file.

18. One or more non-transitory computer readable media according to claim 16, in, The operations further include: collecting sensor data describing an environment surrounding the second computing device; sending sensor data describing the surrounding environment of the second computing device to the server computing system via the application programming interface; and A position of the second computing device relative to a pose of the anchor based on the sensor data is received from the server computing system via the application programming interface.

19. One or more non-transitory computer readable media according to any one of claims 16-18, in, The second computing device performs the operations by implementing a software development kit included in the application.

Images