Position mapping for large scale augmented reality

Abstract

An augmented reality performs operations including accessing a data object comprising image data, position data, and orientation data; applying a transform to the data object to produce a corrected data object; generating a point cloud based on the corrected data object; assigning the point cloud to a location based on at least the position data of the data object; detecting a client device at the location; and in response to detecting the client device at the location, loading the point cloud to the client device.

Description

[0001] Priority requirements

[0002] This application claims priority to U.S. Patent Application No. 16 / 833,160, filed March 27, 2020, which is incorporated herein by reference in its entirety. Technical Field

[0003] Implementations of this disclosure generally relate to mobile computing technologies, and more specifically, but not in a limiting way, to systems for presenting augmented-reality (AR) content at client devices. Background Technology

[0004] Augmented reality (AR) is an interactive experience that uses computer-generated perceptual information to enhance the interaction with real-world environments containing objects, sometimes across multiple perceptual modalities, including visual, auditory, tactile, haptic, and olfactory. The primary value of AR lies in how components of the digital world are integrated into human perception of the real world, rather than as a simple display of data, but through an integration of immersive experiences perceived as a natural part of the environment. Attached Figure Description

[0005] To facilitate the identification of any particular element or action in discussion, one or more of the highest-order digits in the reference numerals indicate the drawing number in which the element was first introduced.

[0006] Figure 1 This is a block diagram illustrating an example messaging system for exchanging data (e.g., messages and related content) over a network according to some embodiments, wherein the messaging system includes an augmented reality system.

[0007] Figure 2 This is a block diagram illustrating further details of a messaging system according to an example implementation.

[0008] Figure 3 This is a block diagram illustrating various modules of an augmented reality system according to some example implementations.

[0009] Figure 4 It is a flowchart depicting a method for generating point clouds according to some example implementations.

[0010] Figure 5 It is a flowchart depicting a method for generating point clouds according to some example implementations.

[0011] Figure 6 This is a flowchart depicting a portion of a method for loading a point cloud at a client device, according to certain example implementations.

[0012] Figure 7This is a diagram illustrating a method for selecting a portion of a point cloud according to certain example implementations.

[0013] Figure 8 This is a block diagram illustrating a representative software architecture that can be used in conjunction with the various hardware architectures described herein and can be used to implement various implementations.

[0014] Figure 9 This is a block diagram illustrating components of a machine, according to some example embodiments, capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and executing any or more of the methods discussed herein. Detailed Implementation

[0015] As mentioned above, augmented reality (AR) is an interactive experience that uses computer-generated perceptual information to enhance the real-world environment of objects existing in the real world. Some AR systems use point clouds to generate and render AR content, where a point cloud is a group of data points in space that measure surface features and the outer surfaces of objects around them.

[0016] Due to the sheer volume of data required to generate point clouds, their use is typically limited to small areas. For instance, creating a point cloud representing surface features within a single room might be relatively straightforward, while creating a point cloud representing surface features of a region or city is logically impossible under current systems for various reasons. Collecting the data needed to generate such large point clouds is inherently cumbersome and time-consuming, requiring extensive organization and analysis. Furthermore, the resulting point clouds generated by such systems would be extremely large and computationally demanding, making them inefficient and impractical for displaying AR content on client devices, including mobile devices.

[0017] Therefore, in some example implementations, an AR system is disclosed that performs the following operations: accessing a data object including image data, location data, and orientation data; and applying a transformation to the data object to produce a corrected data object.

[0018] Point clouds are generated based on calibrated data objects;

[0019] At least based on the location data of the data object, the point cloud is assigned to a location; a client device located at the location is detected; and in response to the detection of a client device located at the location, the point cloud is loaded onto the client device.

[0020] In some example implementations, the data object may include images and videos collected by multiple client devices and indexed within a database based on location data corresponding to the images and videos. The AR system can access the database and generate a point cloud for a given location based on the image data and location data from the images and videos collected from the multiple client devices.

[0021] In some example implementations, the data objects may include images and videos collected from an omnidirectional camera device (360-degree camera device), wherein the 360-degree camera device has a field of view covering at least the entire circle in a substantially sphere or horizontal plane. In such implementations, images and videos may be collected from the 360-degree camera device, wherein the images and videos include timestamps and location data.

[0022] In some example implementations, to generate a point cloud, the AR system can access video data comprising a group of video frames, where each video frame includes a timestamp, location data, orientation data, and image data. The AR system can output a portion of the group of video frames and generate a point cloud based on that portion. To generate a point cloud based on a data object, in some implementations, the AR system can perform a transformation on the data object, where the transformation includes linear correction.

[0023] In some implementations, loading a point cloud at a client device may include identifying a portion of the point cloud to be loaded at the client device. For example, as mentioned above, a technical challenge in rendering AR content using point clouds in a large environment is the computational demands of large-scale point clouds. Therefore, in some implementations, the AR system may identify a portion of the point cloud to be loaded at the client device based on one or more contextual conditions or factors.

[0024] In some implementations, contextual factors may include the location of the client device, where the location and orientation of the client device define the client device's viewpoint. The AR system can determine which landmarks and surface features are visible from the client device's viewpoint and identify a portion of the point cloud based on the landmarks and surface features visible from the client device's viewpoint.

[0025] In some implementations, contextual factors may include the time of day. In such implementations, data objects may include images and videos that include timestamps indicating the time of day in which the images and videos were collected. Therefore, a point cloud may include multiple points, where a single surface feature may be represented by more than one point, and each point may be based on a different time of day. For example, a given surface feature or landmark may have a first set of points representing the surface feature or landmark at a first time of day (i.e., morning), and a second set of points representing the surface feature or landmark at a second time of day (i.e., evening). Thus, an AR system can identify a portion of a point cloud based on temporal considerations, including the time of day when the client device is at a given location or when the client device requests AR content.

[0026] In some implementations, contextual factors may include attributes of the client device itself, such as the client device's memory or storage capacity and network connection speed. Therefore, the optimal size of a portion of the point cloud can be determined based on the client device's device attributes, and a portion of the point cloud can be selected based on this optimal size.

[0027] Figure 1 This is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes one or more client devices 102, each hosting multiple applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104 and the messaging server system 108 via a network 106 (e.g., the Internet).

[0028] Therefore, each messaging client application 104 is able to communicate and exchange data with another messaging client application 104 and with the messaging server system 108 via the network 106. The data exchanged between messaging client applications 104 and between messaging client applications 104 and messaging server system 108 includes functions (e.g., commands to invoke functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0029] The messaging server system 108 provides server-side functionality to a specific messaging client application 104 via network 106. While some functions of the messaging system 100 are described herein as being performed by either the messaging client application 104 or the messaging server system 108, it should be understood that the location of certain functions—whether within the messaging client application 104 or the messaging server system 108—is a design choice. For example, it may technically be preferred that certain technologies and functions be initially deployed within the messaging server system 108, provided the client device 102 has sufficient processing power, but that technology and functions are later migrated to the messaging client application 104.

[0030] The messaging server system 108 supports various services and operations provided to the messaging client application 104. Such operations include sending data to and receiving data from the messaging client application 104, and processing data generated by the messaging client application 104. In some implementations, as examples, this data includes: message content, client device information, geolocation information, media annotations and overlays, message content persistence conditions, social network information, and live event information. In other implementations, other data is used. Data exchange within the messaging system 100 is invoked and controlled via functions available through the GUI of the messaging client application 104.

[0031] Specifically, now turning to message server system 108, application programming interface (API) server 110 is coupled to application server 112 and provides a programming interface to application server 112. Application server 112 is communicatively coupled to database server 118, which provides easy access to database 120, which stores data associated with messages processed by application server 112.

[0032] Specifically, we discuss Application Programming Interface (API) server 110, which receives and sends message data (e.g., commands and message payloads) between client device 102 and application server 112. In particular, API server 110 provides a group of interfaces (e.g., routines and protocols) that messaging client application 104 can invoke or query to call upon or query this group of interfaces to invoke functions of application server 112. Application Programming Interface (API) server 110 exposes various functions supported by application server 112, including: account registration, login functionality, sending messages from a specific messaging client application 104 to another messaging client application 104 via application server 112, sending media files (e.g., images or videos) from messaging client application 104 to messaging server application 114, and possible access by another messaging client application 104, setting up a collection of media data (e.g., stories), retrieving the friend list of the user of client device 102, retrieving such a collection, retrieving messages and content, adding and deleting friends from a social graph, locating friends within a social graph, and opening and applying events (e.g., related to messaging client application 104).

[0033] Application server 112 hosts multiple applications and subsystems, including messaging server application 114, image processing system 116, social networking system 122, and AR system 124. According to some example implementations, AR system 124 is configured to generate a point cloud based on image data and load this point cloud at client device 102. Further details of AR system 124 are available below. Figure 3 Found it.

[0034] The messaging server application 114 implements various messaging techniques and functions, particularly relating to the aggregation and other processing of content (e.g., text and multimedia content) received from multiple instances of the messaging client application 104. As will be described in further detail, text and media content from multiple sources can be aggregated into content collections (e.g., referred to as stories or galleries). The messaging server application 114 then makes these collections available to the messaging client application 104. Considering the hardware requirements for additional processor- and memory-intensive processing of data, such processing can also be performed by the messaging server application 114 on the server side.

[0035] Application server 112 also includes an image processing system 116, which is dedicated to performing various image processing operations typically on images or videos received within the payload of messages at message server application 114.

[0036] Social networking system 122 supports various social networking features and services and makes these features and services available to messaging server application 114. To this end, social networking system 122 maintains and accesses an entity graph within database 120. Examples of features and services supported by social networking system 122 include identifying other users in messaging system 100 who have a relationship with or are “following” a particular user, as well as identifying other entities and interests of a particular user.

[0037] Application server 112 is communicatively coupled to database server 118, which provides easy access to database 120, which stores data associated with messages processed by message server application 114.

[0038] Figure 2 This is a block diagram illustrating further details of a messaging system 100 according to an example embodiment. Specifically, the messaging system 100 is shown as including a messaging client application 104 and an application server 112, which in turn include several subsystems, namely a short-timer system 202, a collection management system 204, and an annotation system 206.

[0039] The short-lived timer system 202 is responsible for implementing temporary access to content permitted by the messaging client application 104 and the messaging server application 114. To this end, the short-lived timer system 202 includes multiple timers that, based on duration and display parameters associated with messages, message sets (e.g., media sets), or graphical elements, selectively display and enable access to messages and associated content via the messaging client application 104. Further details regarding the operation of the short-lived timer system 202 are provided below.

[0040] The collection management system 204 is responsible for managing collections of media (e.g., collections of text, images, video, and audio data). In some examples, collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be made available for a specified time period, such as the duration of the event to which the content relates. For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 204 can also be responsible for publishing icons that provide notifications of the existence of specific collections to the user interface of the messaging client application 104.

[0041] The collection management system 204 also includes a curation interface 208, which enables collection managers to manage and curate specific content collections. For example, the curation interface 208 allows event organizers to curate content collections related to a specific event (e.g., removing inappropriate content or redundant messages). Additionally, the collection management system 204 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some implementations, users may be compensated for including user-generated content in the collection. In such cases, the curation interface 208 operates to automatically pay such users for using their content.

[0042] Annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content associated with a message. For example, annotation system 206 provides functions related to media overlays for generating and publishing messages processed by messaging system 100. Annotation system 206 can operable to provide media overlays (e.g., filters, lenses) to messaging client application 104 based on the geographic location of client device 102. In another example, annotation system 206 can operable to provide media overlays to messaging client application 104 based on other information such as the social network information of the user of client device 102. Media overlays can include audio and visual content and visual effects. Examples of audio and visual content include images, text, logos, animations and sound effects, and animated facial models. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos or videos) at client device 102. For example, media overlays include text that can be overlaid on a photograph generated by client device 102. In another example, media overlays include location identifier overlays (e.g., Venice Beach), live event name overlays, or business name overlays (e.g., Beach Cafe). In yet another example, annotation system 206 uses the geolocation of client device 102 to identify media overlays that include the name of a business at the location of client device 102. Media overlays may include additional tags associated with the business. Media overlays may be stored in database 120 and accessed through database server 118.

[0043] In one example implementation, annotation system 206 provides a user-based publishing platform that allows users to select geographic locations on a map and upload content associated with those locations. Users can also specify the environment in which a particular media overlay should be provided to other users. Annotation system 206 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0044] In another example implementation, annotation system 206 provides a vendor-based publishing platform that enables vendors to select specific media overlays associated with geographic locations via a bidding process. For example, annotation system 206 associates the media overlay of the highest bidder with a corresponding geographic location for a predefined amount of time.

[0045] Figure 3 This is a block diagram illustrating components of an AR system 124 according to certain example embodiments, wherein the components configure the AR system 124 to perform operations to generate and trigger the display of notifications based on categories associated with a user connection.

[0046] AR system 124 is shown as including an image module 302, a correction module 304, and a point cloud module 306, all of which are configured to communicate with each other (e.g., via a bus, shared memory, or switch). Any one or more of these modules can be implemented using one or more processors 308 (e.g., by configuring one or more such processors to perform the functions described for that module) and therefore may include one or more processors 308. In some embodiments, avatar notification system 124 may include a database 120 or be accessible to a database 120, wherein the database 120 may include a collection of media content indexed based on user attributes and astrological symbols.

[0047] Any one or more modules described may be implemented using only hardware (e.g., one or more processors 308 of a machine) or a combination of hardware and software. For example, any module of the described avatar notification system 124 may physically comprise an arrangement of one or more processors 308 (e.g., a subgroup of one or more processors of a machine or one or more processors of a machine) configured to perform the operations described herein for that module. As another example, any module of the avatar notification system 124 may include software, hardware, or both software and hardware configured to perform the operations described herein for that module (e.g., one or more processors of a machine) of an arrangement of one or more processors 308. Thus, different modules of the avatar notification system 124 may include and configure different arrangements of such processors 308 or a single arrangement of such processors 308 at different points in time. Furthermore, any two or more modules of the avatar notification system 124 may be combined into a single module, and the functionality described herein for a single module may be subdivided among multiple modules. Moreover, according to various example embodiments, modules described herein as implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0048] Figure 4This is a flowchart depicting a method 400 for generating point clouds according to certain example implementations. The operation of method 400 can be described above regarding... Figure 3 The module described is used for execution. For example... Figure 4 As shown, method 400 includes one or more operations 402, 404, 406, 408, 410 and 412.

[0049] At operation 402, image module 302 accesses a data object that includes image data, location data, and orientation data. For example, in some embodiments, image module 302 may access a repository (i.e., database 120) containing a collection of location-indexed data objects. For example, in some embodiments, data objects may be collected from multiple client devices 102 and indexed within database 120 based on corresponding location data. In some embodiments, data objects may be generated by an omnidirectional camera and indexed within database 120 based on corresponding location data.

[0050] At operation 404, correction module 304 applies a transformation to the data object to produce a corrected data object. For example, in some embodiments, the data object may include an omnidirectional camera image, wherein the omnidirectional camera image depicts a 360-degree view of the area. Correction module 304 may access the data object and apply one or more linear correction techniques to bring the image into a common image plane.

[0051] At operation 406, point cloud module 306 generates a point cloud based on the corrected data object, wherein the point cloud comprises a group of data points in space that defines the characteristics of the outer surface of the object in space. For example, from a given perspective, the points in the point cloud can define the distance of the surface from that perspective.

[0052] At operation 408, point cloud module 306 assigns the point cloud to a location based at least on the location data of the data object. For example, point cloud module 306 may assign the point cloud to a geofence containing the location. In some implementations, assigning the point cloud to a location may include aligning the point cloud with the location based on landmarks within the location.

[0053] At operation 410, image module 302 detects client device 102 at that location. For example, client device 102 may generate a request including location data identifying the location, or it may input a geofence containing the location. In some implementations, client device 102 may generate image data depicting one or more landmarks associated with the location, and image module 302 may identify the location based on the image data from client device 102.

[0054] At operation 412, in response to the image module 302 detecting the client device 102 at that location, the point cloud module 306 loads the point cloud corresponding to the location of the client device 102. Therefore, AR content can be displayed at the client device 102 based on the point cloud.

[0055] Figure 5 This is a flowchart depicting a method 500 for generating point clouds according to certain example implementations. The operation of method 500 can be described above regarding... Figure 3 The module described is used for execution. For example... Figure 5 As shown, method 500 includes one or more operations 502 and 504 that can be executed as part of operation 402 of method 400 (i.e., subroutine).

[0056] At operation 502, image module 302 accesses video data comprising groups of video frames. For example, the video data may be generated by one or more client devices 102 and indexed in relation to location at memory locations within database 120.

[0057] At operation 504, image module 302 outputs a portion of a group of video frames, wherein each video frame in the portion of the group of video frames includes image data, position data, and orientation data.

[0058] In some implementations, a subset of video frames can be output based on the characteristics of the surface features of the region depicted by the video. For example, a more complex surface may require more video frames to accurately depict the surface as a point cloud, while a simple surface (i.e., few objects and only a few surfaces) may require fewer video frames.

[0059] In some implementations, a portion of a set of video frames may be output based on a collection rate associated with the video data, where the collection rate can be defined with the travel speed. For example, the faster the sensor device moves through an area, the larger the number of video frames may need to be output.

[0060] Therefore, method 500 can continue to operation 404 of method 400, wherein the correction module 304 performs linear correction on the group of video frames to generate a corrected data object.

[0061] Figure 6 This is a flowchart depicting a portion of a method 600 for loading a point cloud at a client device 102 according to certain example implementations. The operation of method 600 can be described above regarding... Figure 3 The module described is used for execution. For example... Figure 6 As shown, method 600 includes one or more operations 602, 604 and 606, and can be performed as part of operation 412 of method 400.

[0062] At operation 602, point cloud module 306 determines the context conditions associated with client device 102. Context conditions may include time conditions (i.e., time of day), device attributes or characteristics, and location data of client device 102.

[0063] For example, in some implementations, context conditions may include the device type of client device 102, the network speed associated with client device 102, and an indication of the memory capacity of client device 102.

[0064] In some implementations, the contextual conditions may include the time of day associated with the request from client device 102. For example, the time of day may be determined based on metadata associated with the request from client device 102 or based on characteristics of an image generated at client device 102.

[0065] In some implementations, the contextual conditions may include the viewpoint of a viewpoint associated with the client device 102, wherein the viewpoint may provide indication of landmarks visible from the location of the client device 102.

[0066] At operation 604, point cloud module 306 identifies a portion of the point cloud associated with that location based on the context conditions of client device 102. At operation 606, a portion of the point cloud is loaded at client device 102.

[0067] Figure 7 This is a diagram 700 depicting a method for selecting a portion of a point cloud according to certain example implementations. Diagram 700 includes depictions of point clouds 705 and 710, where each point cloud represents surface features of the same geographic area.

[0068] As in Figure 6 As discussed in method 600, certain implementations of AR system 124 provide the ability to selectively filter portions of a point cloud to be loaded at client device 102 based on multiple factors, including contextual factors. Thus, point cloud 710 represents a selected portion of points from point cloud 705.

[0069] In some implementations, point cloud module 306 may select a portion of point cloud 705 such that the distribution of points remains the same. For example, as seen in Figure 700, the distribution of points in point cloud 710 is approximately the same as the distribution of points seen in point cloud 705.

[0070] In some implementations, point cloud module 306 can select a portion of point cloud 705 based on the location of user 715. For example, point cloud module 306 can access location data from client device 102 of user 715 and determine visible landmarks and surfaces based on the location of user 715. A portion of point cloud 705 can be selected based on which landmarks are visible.

[0071] In some implementations, point cloud module 306 may select a portion of point cloud 705 based on contextual factors associated with user 715 (i.e., location, time, etc.) and attributes of each point in point cloud 705. For example, points may have attributes indicating the time of day when they were collected. Thus, in response to determining the current time associated with user 715, point cloud module 715 may select from point cloud 705 all points representing surface features of an area at that same time. As an illustrative example, point cloud module 306 may generate point cloud 705 based on data objects including image data, where the image data represents an object or location at a specific time or time of day (i.e., night, day). Therefore, each point in point cloud 705 may include attributes indicating the time of day when that point was collected.

[0072] In some implementations, point cloud module 306 can generate point cloud 710 by starting with an "empty" point cloud and then adding one point at a time from point cloud 705 until a target point cloud characteristic is achieved. For example, the characteristic may include the ratio of visible points to all points as viewed from a given perspective of client device 102. In some implementations, the characteristic may include the size of the point cloud, in bytes.

[0073] Software Architecture

[0074] Figure 8 This is a block diagram illustrating an example software architecture 806, which can be used in conjunction with various hardware architectures described herein. Figure 8 This is a non-limiting example of software architecture, and it should be understood that many other architectures can be implemented to facilitate the functionality described herein. Software Architecture 906 can be applied to, for example... Figure 9 The execution is performed on the hardware of machine 900, which includes processor 904, memory 914, and I / O components 918, etc. A representative hardware layer 852 is shown and can represent, for example... Figure 8 The machine 800. A representative hardware layer 852 includes a processing unit 854 having associated executable instructions 804. The executable instructions 804 represent executable instructions of the software architecture 806, including implementations of the methods, components, etc., described herein. Hardware layer 852 also includes a memory and / or storage module, i.e., a memory / storage device 856, that also has the executable instructions 804. Hardware layer 852 may also include other hardware 858.

[0075] exist Figure 8 In the example architecture, software architecture 806 can be conceptualized as a set of layers, each providing specific functionality. For example, software architecture 806 may include layers such as operating system 802, libraries 820, applications 816, and a presentation layer 814. Operationally, applications 816 and / or other components within a layer can invoke application programming interfaces (APIs) API calls 808 via a software stack and receive responses in response to API calls 808. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile operating systems or dedicated operating systems may not provide a framework / middleware 818, while other operating systems may provide such a layer. Other software architectures may include additional or different layers.

[0076] Operating system 802 can manage hardware resources and provide public services. Operating system 802 may include, for example, a kernel 822, services 824, and drivers 826. Kernel 822 can act as an abstraction layer between hardware and other software layers. For example, kernel 822 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 824 can provide other public services to other software layers. Drivers 826 are responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers 826 may include display drivers, camera drivers, etc. Drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0077] Library 820 provides common infrastructure used by application 816 and / or other components and / or layers. Library 820 provides functionality that allows other software components to perform tasks more easily than by directly interfacing with the functions of the underlying operating system 802 (e.g., kernel 822, services 824, and / or drivers 826). Library 820 may include system libraries 844 (e.g., the C standard library), which provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, library 820 may include API libraries 846 such as media libraries (e.g., libraries supporting the rendering and manipulation of various media formats such as MPREG4, H.264, MP3, AAC, AMR, JPG, and PNG), graphics libraries (e.g., OpenGL frameworks for rendering 2D and 3D content on a display), databases (e.g., SQLite providing various relational database functionalities), network libraries (e.g., WebKit providing web browsing functionality), etc. Library 820 may also include various other libraries 848 to provide many other APIs to application 816 and other software parts / modules.

[0078] The framework / middleware 818 (sometimes also called middleware) provides a higher level of common infrastructure that can be used by applications 816 and / or other software components / modules. For example, the framework / middleware 818 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. The framework / middleware 818 can provide a wide range of other APIs that can be used by applications 816 and / or other software components / modules, some of which may be specific to a particular operating system 802 or platform.

[0079] Application 816 includes built-in applications 838 and / or third-party applications 840. Examples of representative built-in applications 838 may include, but are not limited to: contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 840 may include those used by entities other than the platform-specific vendor using Android. TM or iOS TM Applications developed using a Software Development Kit (SDK) can be used on platforms such as iOS. TM ANDROID TM , Mobile software running on the phone's mobile operating system or other mobile operating systems. Third-party application 840 may call API calls 808 provided by the mobile operating system (e.g., operating system 802) to facilitate the functions described herein.

[0080] Application 816 can use built-in operating system functions (e.g., kernel 822, service 824, and / or driver 826), libraries 820, and frameworks / middleware 818 to create user interfaces to interact with the system's users. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 814. In these systems, the application / component "logic" can be separated from the application / component's user-interactive aspects.

[0081] Figure 9 This is a block diagram illustrating components of a machine 900, according to some example embodiments, capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any or more of the methods discussed herein. Specifically, Figure 9 A graphical representation of a machine 900 in the form of an example computer system is shown, within which instructions 910 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause the machine 900 to perform any or more of the methods discussed herein. Thus, the instructions 910 can be used to implement the modules or components described herein. The instructions 910 transform the general, unprogrammed machine 900 into a specific machine 900 programmed to perform the described and illustrated functions in the described manner. In alternative embodiments, the machine 900 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 900 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 900 may include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart home appliances), other smart devices, web appliances, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 910 specifying actions to be taken by machine 900. Furthermore, although only a single machine 900 is shown, the term "machine" should also be considered to include a collection of machines that individually or jointly execute instructions 910 to perform any one or more of the methods discussed herein.

[0082] Machine 900 may include a processor 904, a memory / storage device 906, and I / O components 918 that can be configured to communicate with each other, for example, via a bus 902. The memory / storage device 906 may include a memory 914, such as main memory or other storage devices, and a storage unit 916, which the processor 904 can access, for example, via the bus 902. Storage units 916 and memory 914 store instructions 910 embodying any one or more of the methods or functions described herein. Instructions 910 may also reside wholly or partially in memory 914, in storage unit 916, in at least one processor of processor 904 (e.g., in the processor's cache memory), or in any suitable combination thereof during execution by machine 900. Therefore, memory 914, storage unit 916, and the memory of processor 904 are examples of machine-readable media.

[0083] I / O component 918 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O component 918 included in a particular machine 900 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It will be understood that I / O component 918 may include... Figure 9 Many other components are not shown. The I / O components 918 are grouped by function only for the purpose of simplifying the following discussion, and this grouping is by no means limiting. In various example embodiments, the I / O components 918 may include output components 926 and input components 928. Output components 926 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input components 928 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), pointing-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide position and / or force for touch or touch gestures), audio input components (e.g., microphones), etc.

[0084] In other example implementations, I / O component 918 may include biometric component 930, motion component 934, environmental component 936, or positioning component 938, as well as various other components. For example, biometric component 930 may include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 934 may include: accelerometer component (e.g., accelerometer), gravity sensor component, rotation sensor component (e.g., gyroscope), etc. Environmental component 936 may include, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), a sound sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of hazardous gases to ensure safety or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 938 may include a position sensor component (e.g., a Global Positioning System (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0085] Various technologies can be used to implement communication. I / O component 918 may include communication component 940, which is operable to couple machine 900 to network 932 or device 920 via coupling 922 and coupling 924, respectively. For example, communication component 940 may include a network interface component or other suitable device to interface with network 932. In other examples, communication component 940 may include wired communication component, wireless communication component, cellular communication component, near field communication (NFC) component, etc. Components (e.g., low power) ), Components and other communication components that provide communication via other modes. Device 920 can be any peripheral device from other machines or various peripheral devices (e.g., peripheral devices coupled via Universal Serial Bus (USB)).

[0086] Furthermore, the communication component 940 can detect identifiers or may include components operable to detect identifiers. For example, the communication component 940 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, data matrices, data symbols, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying audio signals from tags). Additionally, various information can be obtained via the communication component 940, such as location via Internet Protocol (IP) geolocation, etc. Location can be obtained through signal triangulation or by detecting NFC beacon signals that indicate a specific location.

[0087] Glossary

[0088] In this context, "carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions to be executed by a machine, and includes digital or analog communication signals or other intangible media that facilitate the communication of such instructions. Instructions can be sent or received over a network via a network interface device using the transmission medium and employing any of a number of well-known transmission protocols.

[0089] In this context, "client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, laptops, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0090] In this context, "communication network" refers to one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a POTS (Plain Old-Style Telephone Service) network, a cellular telephone network, a wireless network, etc. A network, other types of networks, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, the coupling can implement any of a variety of data transmission technologies, such as Single Carrier Radio Transmission (1xRTT), Evolved Data Optimization (EVDO), General Packet Radio Service (GPRS), Enhanced Data Rate Evolution of GSM (EDGE), the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other standards defined by various standards-setting organizations, other telematics protocols, or other data transmission technologies.

[0091] In this context, a "brief message" refers to a message that can be accessed for a limited time. Brief messages can be text, images, videos, etc. The access time for a brief message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting method, the message is temporary.

[0092] In this context, "machine-readable medium" refers to a component, device, or other tangible medium capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be considered to include a single medium or multiple media capable of storing instructions (e.g., a centralized or distributed database or associated cache memory and server). The term "machine-readable medium" should also be considered to include any medium or combination of media capable of storing machine-executable instructions (e.g., code) such that, when executed by one or more processors of the machine, the instructions cause the machine to perform any or more of the methods described herein. Therefore, "machine-readable medium" refers to a single storage device or apparatus, and a "cloud-based" storage system or storage network comprising multiple storage devices or apparatuses. The term "machine-readable medium" does not include signals themselves.

[0093] In this context, a “component” refers to a device, physical entity, or logic having boundaries defined by functional or subroutine calls, branch points, application programming interfaces (APIs), or other technologies that provide partitioning or modularity for a particular processing or control function. A component can be combined with other components via its interface to perform machine processing. A component can be an encapsulated functional hardware unit designed for use with other components and can be part of a program that typically performs a specific function within a related function. A component can constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various example implementations, one or more hardware components (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to perform certain operations described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include dedicated circuitry or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuitry temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific part of a machine) uniquely tailored to perform the configured function, and no longer a general-purpose processor. It will be understood that the hardware component may be implemented mechanically in a dedicated and permanently configured circuit or in a temporarily configured (e.g., software-configured) circuit, for cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Considering the implementation where hardware components are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any given time. For example, where the hardware components include a general-purpose processor that can be configured as a dedicated processor via software, the general-purpose processor can be configured as different dedicated processors (e.g., including different hardware components) at different times. The software accordingly configures one or more specific processors to constitute a specific hardware component at one time and different hardware components at different times. Hardware components can provide information to and receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In embodiments where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessed by the multiple hardware components and retrieving information from that memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Other hardware components can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., information collection). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily or permanently configured (e.g., via software) to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by processors, where a particular processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or processor-implemented components. Furthermore, one or more processors can also operate to support the execution of related operations in a “cloud computing” environment or as a “Software as a Service” (SaaS) operation. For example, at least some operations of the operations can be performed by a group of computers (as an example of a machine including processors), where these operations are accessible via a network (e.g., the Internet) and via one or more suitable interfaces (e.g., application programming interfaces (APIs)). The execution of some operations of the operations can be distributed among processors, not only residing within a single machine but also deployed across multiple machines. In some example implementations, the processor or processor-implemented component may reside in a single geographic location (e.g., within a home environment, office environment, or server cluster). In other example implementations, the processor or processor-implemented component may be distributed across multiple geographic locations.

[0094] In this context, "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and generates corresponding output signals that are applied to operate the machine. For example, a processor can be a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Radio Frequency Integrated Circuit (RFIC), or any combination thereof. A processor can also be a multi-core processor having two or more independent processors (sometimes called "cores") capable of executing instructions simultaneously.

[0095] In this context, a "timestamp" refers to a sequence of characters or encoded information that identifies when an event occurred, such as giving a date and time of day, sometimes accurate to a fraction of a second.

[0096] In this context, "LIFT" is a measure of the performance of a randomly selected targeted model in predicting or classifying cases as having an enhanced response (relative to the entire population).

[0097] In this context, "phoneme alignment" refers to the fact that a phoneme is a phonological unit that distinguishes one word from another. A phoneme can consist of a series of closure, burst, and aspiration events; or, a diphthong can transition from a back vowel to a front vowel. Therefore, a speech signal can be described not only by the phonemes it contains but also by the positions of those phonemes. Thus, phoneme alignment can be described as the "temporal alignment" of phonemes in a waveform to determine the appropriate sequence and position of each phoneme in the speech signal.

[0098] In this context, "audio-to-visual conversion" refers to converting audible speech signals into visual speech, where visual speech can include mouth shapes representing audible speech signals.

[0099] In this context, "Time Delayed Neural Network (TDNN)" refers to an artificial neural network architecture whose primary purpose is to process sequential data. An example is converting continuous audio into a stream of categorized phoneme labels for speech recognition.

[0100] In this context, "Bidirectional Long Short-Term Memory (BLSTM)" refers to a recurrent neural network (RNN) architecture that stores values ​​over arbitrary intervals. The stored values ​​are not modified as learning progresses. RNNs allow for forward and backward connections between neurons. Given unknown time delays and the duration between events, BLSTMs are well-suited for classifying, processing, and predicting time series data.

Claims

1. A method for augmented reality, comprising: Access data objects that include image data, location data, and orientation data; Perform linear correction on the data object to produce a corrected data object; A point cloud is generated based on the corrected data object; The point cloud is assigned to a location based at least on the location data of the data object; Detect the client device located at the specified location; Determine contextual conditions associated with the client device, the contextual conditions including at least one of the client device's location and orientation that define the client device's viewpoint; Identifying a portion of the point cloud based on the contextual conditions, wherein identifying a portion of the point cloud includes determining landmarks and surface features visible from the viewpoint of the client device; as well as The portion of the point cloud is loaded into the client device.

2. The method according to claim 1, wherein, The data object includes video frames, and accessing the video frames includes: Access video data including groups of video frames; and Output a portion of the group of video frames, wherein the portion of the group of video frames includes the video frames.

3. The method according to claim 1, wherein, The data objects include data from 360-degree camera devices.

4. The method according to claim 1, wherein, Detecting the client device at the location includes: Receive a request from the client device that includes an identifier of the location.

5. The method according to claim 1, wherein, The method further includes: This enables the display of media content on the client device based on the point cloud.

6. The method according to claim 1, wherein, The contextual conditions include the connection speed of the network associated with the client device.

7. The method according to claim 1, wherein, The contextual conditions include the time of day.

8. The method according to claim 1, wherein, The context conditions include user profile data.

9. The method according to claim 1, wherein, Loading the point cloud to the client device includes: Identify landmarks based on the image data of the data object; Identify a portion of the point cloud based on the landmarks; and A portion of the point cloud is loaded at the client device.

10. A system for augmented reality, comprising: Memory; as well as At least one hardware processor coupled to the memory and including instructions to cause the system to perform operations, said operations including: Access data objects that include image data, location data, and orientation data; Perform linear correction on the data object to produce a corrected data object; A point cloud is generated based on the corrected data object; The point cloud is assigned to a location based at least on the location data of the data object; Detect the client device located at the specified location; Determine contextual conditions associated with the client device, the contextual conditions including at least one of the client device's location and orientation that define the client device's viewpoint; Identifying a portion of the point cloud based on the contextual conditions, wherein identifying a portion of the point cloud includes determining landmarks and surface features visible from the viewpoint of the client device; and The portion of the point cloud is loaded into the client device.

11. The system according to claim 10, wherein, The data object includes video frames, and accessing the video frames includes: Access video data including groups of video frames; and Output a portion of the group of video frames, wherein the portion of the group of video frames includes the video frames.

12. The system according to claim 10, wherein, The data objects include data from 360-degree camera devices.

13. The system according to claim 10, wherein, Detecting the client device at the location includes: Receive a request from the client device that includes an identifier of the location.

14. The system according to claim 10, wherein, The operation also includes: This enables the display of media content on the client device based on the point cloud.

15. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of the machine, cause the machine to perform operations, the operations including: Access data objects that include image data, location data, and orientation data; Perform linear correction on the data object to produce a corrected data object; A point cloud is generated based on the corrected data object; The point cloud is assigned to a location based at least on the location data of the data object; Detect the client device located at the specified location; Determine contextual conditions associated with the client device, the contextual conditions including at least one of the client device's location and orientation that define the client device's viewpoint; Identifying a portion of the point cloud based on the contextual conditions, wherein identifying a portion of the point cloud includes determining landmarks and surface features visible from the viewpoint of the client device; as well as The portion of the point cloud is loaded into the client device.

16. The non-transitory machine-readable storage medium according to claim 15, wherein, The data object includes video frames, and accessing the video frames includes: Access video data including groups of video frames; and Output a portion of the group of video frames, wherein the portion of the group of video frames includes the video frames.

Images