Virtual surface modification

Abstract

Aspects of the present disclosure relate to a system comprising a computer-readable storage medium storing at least one program and a method for rendering a virtual modification of a real-world environment depicted in image content. A reference surface is detected in a three-dimensional (3D) space captured within a camera feed generated by a camera of a computing device. An image mask is applied to the reference surface within the 3D space captured within the camera feed. A visual effect is applied to the image mask corresponding to the reference surface in the 3D space. Applying the visual effect to the image mask results in the modified surface being rendered in the camera feed being presented on a display of the computing device.

Description

Technical Field

[0001] This application claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Application Serial No. 62 / 782,916, filed on December 20, 2018, entitled “Virtual Surface Modification,” which is incorporated herein by reference. Background Art

[0002] The present disclosure relates generally to visual presentation and, more particularly, to rendering virtual modifications to real-world environments depicted in a camera feed. Summary of the Invention

[0003] Virtual rendering systems can be used to create engaging and entertaining augmented reality experiences in which three-dimensional (3D) virtual object graphical content appears to exist in the real world. Such systems can experience rendering issues due to environmental conditions, user motion, unexpected visual interruptions between the camera and the object being rendered, and so on. This can cause virtual objects to disappear or otherwise behave erratically, breaking the illusion of virtual objects existing in the real world. For example, as a user moves through the real world, a virtual rendering system may not render virtual objects consistently relative to real-world items. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, one or more of the most significant digits in a reference number refers to the figure number in which the element is first introduced. Some embodiments are illustrated by way of example and not limitation in the accompanying drawings, in which:

[0005] Figure 1 is a block diagram illustrating an example messaging system for exchanging data (e.g., messages and associated content) over a network according to an example embodiment;

[0006] Figure 2 is a block diagram illustrating further details regarding a messaging system according to an example embodiment;

[0007] Figure 3 is a schematic diagram illustrating data that may be stored in a database of a messaging server system according to an example embodiment;

[0008] Figure 4 is a schematic diagram illustrating the structure of a message generated by a messaging client application for communication according to an example embodiment;

[0009] Figure 5is a diagram illustrating an example access restriction process in which access to content (e.g., an ephemeral message and associated multimedia data payload) or a collection of content (e.g., an ephemeral message story) may be time-restricted (e.g., made ephemeral) according to an example embodiment;

[0010] Figure 6 is a block diagram illustrating various components of a virtual rendering system according to an example embodiment;

[0011] Figure 7 is a flowchart illustrating example operations of a virtual rendering system in performing a method for rendering a virtual modification into a 3D space according to an example embodiment;

[0012] Figure 8 is a flowchart illustrating example operations of a virtual rendering system in performing a method for tracking an object rendered in a 3D space according to an example embodiment;

[0013] Figure 9-13 is a flowchart illustrating example operations of a virtual rendering system in performing a method for rendering a virtual modification to a surface in a 3D space according to an example embodiment;

[0014] Figure 14A and 14B is an interface diagram illustrating an interface provided by a virtual rendering system according to an example embodiment;

[0015] Figure 15 is a block diagram illustrating a representative software architecture that may be used in conjunction with the various hardware architectures described herein, according to an example embodiment; and

[0016] Figure 16 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to example embodiments. DETAILED DESCRIPTION

[0017] The following description includes systems, methods, techniques, instruction sequences, and computer program products that embody the illustrative embodiments of the present disclosure. In the following description, for the purpose of explanation, many specific details are set forth to provide an understanding of the various embodiments of the subject matter of the present invention. However, it will be apparent to those skilled in the art that embodiments of the subject matter of the present invention can be practiced without these specific details. Generally speaking, well-known instruction instances, protocols, structures, and techniques need not be shown in detail.

[0018] Among other things, embodiments of the present disclosure improve the functionality of electronic messaging and imaging software and systems by rendering virtual modifications to a 3D real-world environment depicted in image data (e.g., images and videos) as if the modifications existed in the real-world environment. For example, the system can render one or more visual effects applied to a real-world surface within a 3D space depicted in image content generated by an image capture device (e.g., a digital camera). The one or more visual effects can be rendered so that the modified surface appears to exist in the real-world environment. The visual effect applied to the real-world surface can be any of a variety of visual effects, including, for example, changing the color of the surface, changing the texture of the surface, applying animation effects to the surface (e.g., flowing water), blurring the surface, rendering a moving virtual object whose movement is constrained by the boundaries of the surface, replacing the surface with other visual content, and various combinations thereof.

[0019] Figure 1 1 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. Messaging system 100 includes multiple client devices 102, each hosting multiple applications, including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of messaging client applications 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0020] Thus, each messaging client application 104 is able to communicate and exchange data with another messaging client application 104 and the messaging server system 108 over the network 106. The data exchanged between the messaging client applications 104 and between the messaging client applications 104 and the messaging server system 108 includes functions (e.g., commands to invoke functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0021] The messaging server system 108 provides server-side functionality to specific messaging client applications 104 over the network 106. Although certain functions of the messaging system 100 are described herein as being performed by either the messaging client application 104 or the messaging server system 108, it should be understood that the location of certain functions within the messaging client application 104 or certain functions of the messaging server system 108 is a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the messaging server system 108, but later migrate the technologies and functions to the messaging client application 104 if the client device 102 has sufficient processing power.

[0022] The messaging server system 108 supports various services and operations provided to the messaging client applications 104. Such operations include sending data to the messaging client applications 104, receiving data from them, and processing data generated by them. By way of example, this data may include message content, client device information, geolocation information, media annotations and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the messaging system 100 is invoked and controlled through functionality available through the user interface (UI) of the messaging client applications 104.

[0023] Turning now specifically to the messaging server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

[0024] Specifically, the API server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the API server 110 provides an interface (e.g., routines and protocols) that can be called or queried by the messaging client application 104 to invoke functionality of the application server 112. The API server 110 exposes various functionality supported by the application server 112, including account registration, login functionality, sending messages from a particular messaging client application 104 to another messaging client application 104 via the application server 112, sending media files (e.g., images or videos) from a messaging client application 104 to a messaging server application 114 for possible access by another messaging client application 104, setting up media data collections (e.g., stories), retrieving such collections, retrieving a friend list of the user of the client device 102, retrieving messages and content, adding and removing friends from a social graph, the location of friends within the social graph, and open application events (e.g., related to the messaging client application 104).

[0025] The application server 112 hosts a number of applications and subsystems, including a messaging server application 114, an image processing system 116, and a social networking system 122. The messaging server application 114 implements a variety of message processing techniques and functions, particularly relating to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client application 104. As will be described in further detail, text and media content from multiple sources can be aggregated into content collections (e.g., referred to as stories or galleries). The messaging server application 114 then makes these collections available to the messaging client application 104. Given the hardware requirements of other processor- and memory-intensive data processing, such processing can also be performed on the server side by the messaging server application 114.

[0026] As will be discussed below, the messaging server application 114 includes a virtual rendering system that provides functionality for generating, rendering, and tracking visual modifications within a 3D real-world environment depicted in the camera view of the client device 102 .

[0027] The application server 112 also includes an image processing system 116 dedicated to performing various image processing operations, typically with respect to images or videos received within the payload of messages at the messaging server application 114 .

[0028] The social networking system 122 supports various social networking features and services and makes these features and services available to the messaging server application 114. To do so, the social networking system 122 maintains and accesses an entity graph within the database 120. Examples of features and services supported by the social networking system 122 include identification of other users of the messaging system 100 with whom a particular user has relationships or is "following," as well as identification of other entities and interests of the particular user.

[0029] The application server 112 is communicatively coupled to a database server 118 , which facilitates access to a database 120 where data associated with messages processed by the messaging server application 114 is stored.

[0030] Figure 2 is a block diagram illustrating further details regarding messaging system 100 according to an example embodiment. Specifically, messaging system 100 is shown as including messaging client application 104 and application server 112, which in turn embodies a plurality of subsystems, namely, transient timer system 202, collection management system 204, annotation system 206, and virtual rendering system 210.

[0031] The transient timer system 202 is responsible for enforcing temporary access to content permitted by the messaging client application 104 and the messaging server application 114. To this end, the transient timer system 202 includes a plurality of timers that selectively display and enable access to messages and associated content via the messaging client application 104 based on duration and display parameters associated with the message or collection of messages. Further details regarding the operation of the transient timer system 202 are provided below.

[0032] The collection management system 204 is responsible for managing media collections (e.g., collections of text, images, video, and audio data). In some examples, collections of content (e.g., messages, which include images, videos, text, and audio) can be organized into "event libraries" or "event stories." Such collections can be available for a specified period of time (such as the duration of the event to which the content relates). For example, content related to a concert can be available as a "story" for the duration of the concert. The collection management system 204 can also be responsible for publishing an icon to the user interface of the messaging client application 104 that provides notification of the existence of a particular collection.

[0033] The collection management system 204 also includes a curation interface 208 that allows collection managers to manage and curate specific content collections. For example, the curation interface 208 enables event organizers to curate content collections related to a specific event (e.g., removing inappropriate content or redundant messages). In addition, the collection management system 204 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some embodiments, users can be compensated for including user-generated content in a collection. In this case, the curation interface 208 operates to automatically pay these users for use of their content.

[0034] The annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content associated with messages. For example, the annotation system 206 provides functions related to the generation and publication of media overlays for messages processed by the messaging system 100. The annotation system 206 is operable to provide media overlays (e.g., filters) to the messaging client application 104 based on the geographic location of the client device 102. In another example, the annotation system 206 is operable to provide media overlays to the messaging client application 104 based on other information, such as social network information of the user of the client device 102. Media overlays can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos) at the client device 102. For example, a media overlay includes text that can be overlaid on top of a photo generated by the client device 102. In another example, a media overlay includes the identification of a location overlay (e.g., Venice Beach), the name of a live event, or the name of a business overlay (e.g., Beach Cafe). In another example, the annotation system 206 uses the geolocation of the client device 102 to identify a media overlay that includes the name of a business at the geolocation of the client device 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 120 and accessed through the database server 118.

[0035] In one example embodiment, the annotation system 206 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. Users can also specify circumstances under which specific content should be provided to other users. The annotation system 206 generates a media overlay including the uploaded content and associates the uploaded content with the selected geographic location.

[0036] In another example embodiment, the annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media coverage associated with a geographic location through a bidding process. For example, the annotation system 206 associates the media coverage of the highest bidding merchant with the corresponding geographic location for a predefined amount of time.

[0037] The virtual rendering system 210 provides functionality for generating, rendering, and tracking virtual modifications within the 3D real-world environment depicted in the live camera feed (also referred to by those of ordinary skill in the art as a "camera stream," "video stream," or "video source") of the client device 102. The virtual modifications provided by the virtual rendering system 210 may include applying one or more visual effects to the real-world surfaces depicted in the camera feed. The virtual modifications provided by the virtual rendering system 210 may also include virtual objects rendered within the real-world environment depicted in the live camera feed of the client device 102.

[0038] Figure 3 is a schematic diagram illustrating data 300 that may be stored in database 120 of messaging server system 108, according to certain example embodiments. Although the contents of database 120 are shown as including a plurality of tables, it should be understood that the data may be stored in other types of data structures (e.g., as an object-oriented database).

[0039] The database 120 includes message data stored in a message table 314. The entity table 302 stores entity data, including an entity graph 304. The entities whose records are maintained in the entity table 302 may include individuals, corporate entities, organizations, objects, places, etc. Regardless of the type, any entity about which the messaging server system 108 stores data may be an identified entity. Each entity is provided with a unique identifier, as well as an entity type identifier (not shown).

[0040] The entity graph 304 also stores information about relationships and associations between entities. For example, such relationships can be social, professional (e.g., working in a common company or organization), interest-based, or activity-based.

[0041] The database 120 also stores annotation data in the form of filters and lenses, for example, in the annotation table 312. The filters and lenses for which data is stored in the annotation table 312 are associated with and applied to videos (whose data is stored in the video table 310) and / or images (whose data is stored in the image table 308). A filter is an overlay that is displayed as an overlay on an image or video during presentation to a receiving user. A lens comprises real-time visual effects and / or sounds that can be added to a camera feed to depict a real-world environment, for example (when a user is viewing the camera feed via one or more interfaces of the messaging client application 104, while composing a message, or during presentation to a receiving user). In some embodiments, a filter is applied to an image or video, while a lens is applied to the camera feed of the client device 102, after the image or video is captured at the client device 102, such that when the image or video is captured at the client device 102 with the lens applied, the applied lens is incorporated as part of the generated image or video. Filters and lenses can be of various types, including user-selected filters and lenses from a filter gallery or lens gallery that are presented to the sending user by the messaging client application 104 while the sending user is composing a message.

[0042] As described above, video table 310 stores video data that, in one embodiment, is associated with messages for which records are maintained in message table 314. Similarly, image table 308 stores image data associated with messages for which message data is stored in entity table 302. Entity table 302 may associate various annotations from annotation table 312 with various images and videos stored in image table 308 and video table 310.

[0043] The story table 306 stores data about a collection of messages and associated images, videos, or audio data, compiled into a collection. The creation of a particular collection can be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 302). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by that user. To this end, the user interface of the messaging client application 104 may include a user-selectable icon that enables the sending user to add specific content to his or her personal story.

[0044] A collection may also constitute a "Live Story," which is a collection of content from multiple users that is created manually, automatically, or using a combination of manual and automatic techniques. For example, a "Live Story" may constitute a curated stream of user-submitted content from various locations and events. For example, a user whose client device has location services enabled and who is at a public location event at a particular time may be presented with the option to contribute content to a particular Live Story via the user interface of the messaging client application 104. The Live Story may be identified to the user by the messaging client application 104 based on his or her location. The end result is a "Live Story" told from the perspective of the community.

[0045] Another type of content collection is called a "location story," which enables users whose client devices 102 are located in a particular geographic location (e.g., on a college or university campus) to contribute to a particular collection. In some embodiments, contributions to location stories may require a second degree of authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0046] Figure 4 is a schematic diagram illustrating the structure of a message 400 generated by a messaging client application 104 for communication with another messaging client application 104 or a messaging server application 114, according to some embodiments. The content of a particular message 400 is used to populate a message table 314 stored within a database 120, which is accessible by a messaging server application 114. Similarly, the content of the message 400 is stored in memory as "in-flight" or "in-transit" data of a client device 102 or application server 112. The message 400 is shown as including the following components:

[0047] Message identifier 402 : A unique identifier that identifies the message 400 .

[0048] Message text payload 404 : text generated by the user via the user interface of the client device 102 and included in the message 400 .

[0049] Message image payload 406 : Image data captured by a camera component of the client device 102 or retrieved from the memory of the client device 102 and included in the message 400 .

[0050] Message video payload 408 : Video data, captured by a camera component or retrieved from a memory component of the client device 102 and included in the message 400 .

[0051] Message audio payload 410 : audio data captured by a microphone or retrieved from a memory component of the client device 102 and included in the message 400 .

[0052] Message annotation 412: Annotation data (eg, a filter, sticker, or other enhancement) representing an annotation to be applied to the message image payload 406, the message video payload 408, or the message audio payload 410 of the message 400.

[0053] Message duration parameter 414: A parameter value that indicates, in seconds, the amount of time that the message content (e.g., message image payload 406, message video payload 408, message audio payload 410) will be presented to or accessible to the user via the messaging client application 104.

[0054] Message geolocation parameters 416: Geolocation data (e.g., latitude and vertical coordinates) associated with the content payload of the message 400. Multiple message geolocation parameter 416 values may be included in the payload, with each of these parameter values being associated with a content item included in the content (e.g., a specific image within the message image payload 406, or a specific video within the message video payload 408).

[0055] Message story identifier 418: An identifier value that identifies one or more content collections (e.g., "story") associated with a particular content item in the message image payload 406 of the message 400. For example, multiple images within the message image payload 406 may each be associated with multiple content collections using an identifier value.

[0056] Message Tags 420: Each message 400 may be tagged with a plurality of tags, each of which indicates the subject matter of the content included in the message payload. For example, if a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value may be included within the message tags 420 that indicates the relevant animal. Tag values may be manually generated based on user input, or may be automatically generated using, for example, image recognition.

[0057] • Message sender identifier 422: An identifier (eg, a messaging system identifier, an email address, or a device identifier) that indicates the user of the client device 102 on which the message 400 was generated and from which it was sent.

[0058] • Message recipient identifier 424: An identifier (eg, a messaging system identifier, an email address, or a device identifier) that indicates the user of the client device 102 to which the message 400 is addressed.

[0059] The content (e.g., value) of the various components of message 400 may be pointers to locations in a table where the content data values are stored. For example, the image value in message image payload 406 may be a pointer to a location (or its address) within image table 308. Similarly, the value within message video payload 408 may point to data stored within video table 310, the value stored within message annotation 412 may point to data stored in annotation table 312, the value stored within message story identifier 418 may point to data stored in story table 306, and the values stored within message sender identifier 422 and message recipient identifier 424 may point to user records stored within entity table 302.

[0060] Figure 5 is a schematic diagram illustrating an access restriction process 500 in which access to content (e.g., an ephemeral message 502 and an associated multimedia payload of data) or a collection of content (e.g., an ephemeral message story 504) can be time-limited (e.g., made ephemeral).

[0061] Ephemeral message 502 is shown associated with a message duration parameter 506, the value of which determines the amount of time that messaging client application 104 will display ephemeral message 502 to the receiving user of ephemeral message 502. In one embodiment, where messaging client application 104 is an application client, ephemeral message 502 may be viewed by the receiving user for up to 10 seconds, depending on the amount of time specified by the sending user using message duration parameter 506.

[0062] The message duration parameter 506 and the message recipient identifier 424 are shown as inputs to a message timer 512, which is responsible for determining the amount of time that the ephemeral message 502 is displayed to the particular receiving user identified by the message recipient identifier 424. Specifically, the ephemeral message 502 will only be displayed to the associated receiving user for a period of time determined by the value of the message duration parameter 506. The message timer 512 is shown as providing an output to the more general ephemeral timer system 202, which is responsible for the overall timing of displayed content (e.g., the ephemeral message 502) for the receiving user.

[0063] Short message 502 Figure 5100 is shown as being included within an ephemeral message story 504 (e.g., a personal story or an event story). The ephemeral message story 504 has an associated story duration parameter 508, the value of which determines the duration for which the ephemeral message story 504 is presented to and accessible to users of the messaging system 100. For example, the story duration parameter 508 may be the duration of a concert, wherein the ephemeral message story 504 is a collection of content related to the concert. Alternatively, the user (the owning user or the curating user) may specify the value of the story duration parameter 508 when performing the setup and creation of the ephemeral message story 504.

[0064] In addition, each ephemeral message 502 within the ephemeral message story 504 has an associated story engagement parameter 510, the value of which determines the duration for which the ephemeral message 502 will be accessible within the context of the ephemeral message story 504. Thus, a particular ephemeral message story 504 may "expire" and become inaccessible within the context of the ephemeral message story 504 before the ephemeral message story 504 itself expires in terms of the story duration parameter 508. The story duration parameter 508, the story engagement parameter 510, and the message recipient identifier 424 each provide input to a story timer 514, which first operatively determines whether a particular ephemeral message 502 of the ephemeral message story 504 will be displayed to a particular receiving user, and if so, for how long. Note that, as a result of the message recipient identifier 424, the ephemeral message story 504 is also aware of the identity of the particular receiving user.

[0065] Thus, the story timer 514 is operable to control the overall lifespan of the associated ephemeral message story 504, as well as the individual ephemeral messages 502 included in the ephemeral message story 504. In one embodiment, each ephemeral message 502 within the ephemeral message story 504 remains visible and accessible for the period of time specified by the story duration parameter 508. In another embodiment, a particular ephemeral message 502 may expire based on the story engagement parameter 510 within the context of the ephemeral message story 504. Note that even within the context of the ephemeral message story 504, the message duration parameter 506 may still determine the duration for which a particular ephemeral message 502 is displayed to the receiving user. Thus, the message duration parameter 506 determines the duration for which a particular ephemeral message 502 is displayed to the receiving user, regardless of whether the receiving user is viewing the ephemeral message 502 within or outside the context of the ephemeral message story 504.

[0066] The ephemeral timer system 202 may also be operable to remove a particular ephemeral message 502 from the ephemeral message story 504 based on a determination that the particular ephemeral message 502 has exceeded the associated story engagement parameter 510. For example, when the sending user has established a story engagement parameter 510 of 24 hours from posting, the ephemeral timer system 202 will remove the associated ephemeral message 502 from the ephemeral message story 504 after the designated 24 hours. The ephemeral timer system 202 is also operable to remove an ephemeral message story 504 when the story engagement parameter 510 of each ephemeral message 502 within the ephemeral message story 504 has expired, or when the ephemeral message story 504 itself has expired in terms of the story duration parameter 508.

[0067] In some use cases, the creator of a particular ephemeral message story 504 may specify an indefinite story duration parameter 508. In this case, the expiration of the story engagement parameter 510 for the last remaining ephemeral message 502 within the ephemeral message story 504 will determine when the ephemeral message story 504 itself expires. In this case, a new ephemeral message 502 with a new story engagement parameter 510 added to the ephemeral message story 504 effectively extends the lifespan of the ephemeral message story 504 to a value equal to the story engagement parameter 510.

[0068] In response to the ephemeral timer system 202 determining that the ephemeral message story 504 has expired (e.g., is no longer accessible), the ephemeral timer system 202 communicates with the messaging system 100 (and, e.g., specifically, the messaging client application 104) to cause the indicia (e.g., an icon) associated with the relevant ephemeral message story 504 to no longer be displayed within the user interface of the messaging client application 104. Similarly, when the ephemeral timer system 202 determines that the message duration parameter 506 of a particular ephemeral message 502 has expired, the ephemeral timer system 202 causes the messaging client application 104 to no longer display the indicia (e.g., an icon or textual identification) associated with the ephemeral message 502.

[0069] Figure 6 2 is a block diagram illustrating functional components of a virtual rendering system 210 that is configured to render virtual modifications to a 3D real-world environment depicted in a live camera feed. For example, the virtual rendering system 210 may render virtual modifications to real-world surfaces in a 3D space depicted in a live camera feed. As another example, the virtual rendering system 210 may render virtual objects within the 3D space.

[0070] The virtual rendering system 210 is shown to include a rendering component 602, a tracking system 604, an interruption detection component 606, an object template component 608, and an event detection component 610. The various components of the virtual rendering system 210 can be configured to communicate with each other (e.g., via a bus, shared memory, or switch). Figure 6 However, in some embodiments, the virtual rendering system 210 may include or may be in communication with a camera configured to generate a camera feed including image data including a sequence of images (eg, a video).

[0071] Any one or more of the components described may be implemented using hardware alone (e.g., one or more processors 612 of a machine) or a combination of hardware and software. For example, any component described by the virtual rendering system 210 may physically include an arrangement of one or more of the processors 612 (e.g., a subset of the one or more processors of the machine or a subset thereof), the processors 612 being configured to perform the operations described herein. As another example, any component of the virtual rendering system 210 may include software, hardware, or both that configures an arrangement of one or more processors 612 (e.g., among the one or more processors of the machine) to perform the operations described herein. Thus, different components of the virtual rendering system 210 may include and configure different arrangements of such processors 612 or a single arrangement of such processors 612 at different points in time.

[0072] In addition, any two or more components of the virtual rendering system 210 can be combined into a single component, and the functionality described herein for a single component can be subdivided among multiple components. In addition, according to various example embodiments, components described herein as being implemented within a single machine, database, or device can be distributed across multiple machines, databases, or devices.

[0073] Tracking system 604 may include a first tracking subsystem 604A, a second tracking subsystem 604B, and a third tracking subsystem 604C. Each tracking subsystem tracks the position of a virtual modification to the 3D space based on a set of tracking markers.

[0074] Tracking systems often experience tracking failures due to environmental conditions, user motion, unexpected visual interruptions between the camera and the tracked object / scene, and more. Traditionally, such tracking failures result in a disruption in the rendering of virtual objects in 3D space. For example, virtual objects may disappear or otherwise behave erratically, disrupting the illusion of the virtual object being within 3D space. This can undermine the perceived quality of the overall 3D experience.

[0075] Traditional tracking systems rely on a single method (natural feature tracking (NFT), simultaneous localization and mapping (SLAM), gyroscopes, etc.), each of which has breakpoints in practical use due to inaccurate sensor data, movement, loss or occlusion of visual markers, or dynamic interruptions in the scene. In addition, each method may have its own limitations in capabilities. For example, a gyroscope tracking system can only track items with three degrees of freedom (3DoF). In addition, due to the inherent limitations of each individual system, using a single tracking system can provide inaccurate or unstable position estimates. For example, due to the inaccuracies of visual tracking alone, an NFT system may not provide adequate pitch, yaw, or roll estimates, while a gyroscope tracking system provides inaccurate translation (up, down, left, right).

[0076] To address the aforementioned issues with conventional tracking systems, the virtual rendering system 210 includes multiple redundant tracking subsystems 604A-C that enable seamless transitions between tracking subsystems. The multiple redundant tracking subsystems 604A-C address the issues with conventional tracking systems by merging multiple tracking methods into a single tracking system 604. The tracking system 604 is capable of combining 6DoF and 3DoF tracking technologies by combining and transitioning between the multiple tracking systems based on the availability of tracking markers tracked by the tracking systems. Therefore, when a marker tracked by any one tracking system becomes unavailable, the virtual rendering system 210 seamlessly switches between tracking in 6DoF and 3DoF, providing the user with an uninterrupted experience. For example, in the case of a visual tracking system (e.g., NFT, SLAM), the tracking markers that are typically analyzed to determine orientation can be replaced with gyroscopic tracking markers from a gyroscopic tracking system. This enables transitions between tracking in 6DoF and 3DoF based on the availability of tracking markers.

[0077] In some example embodiments, to switch between tracking in 6DoF and 3DoF, the virtual rendering system 210 collects and stores tracking markers within a tracking matrix that includes translation markers (e.g., up, down, left, right) and rotation markers (e.g., pitch, yaw, roll). Thus, the translation markers collected by the NFT system can be extracted from the tracking matrix and used when future translation markers collected by the NFT system become inaccurate or unavailable. At the same time, the gyroscope continues to provide rotation markers. In this way, when the mobile device loses the tracking marker, the tracked object presented in 3D space is not abruptly changed at the frame when the tracking marker is lost. Subsequently, when the target tracked object reappears on screen and acquires a new translation T1, the translation portion of the view matrix will utilize the new translation T1 and use T1-T0 as the translation of the view matrix.

[0078] The rendering component 602 of the virtual rendering system 210 is configured to render virtual modifications in the 3D space captured within the live camera feed generated by the camera of the client device 102. For example, the rendering component 602 can render visual effects applied to real-world surfaces in the 3D space captured within the live camera feed. When rendering the modifications, the virtual rendering system 210 dynamically applies an image mask to the surface depicted in the live camera feed and applies the visual effect to the image mask.

[0079] The virtual rendering system 210 can track and adjust the position of the virtual modification using one or more tracking systems in accordance with 6DoF. For example, one or more tracking systems of the virtual rendering system 210 can collect and analyze a set of tracking markers (e.g., roll, pitch, yaw, natural features, etc.) to track the position of the virtual modification relative to the client device 102 in a 3D space with 6DoF. In such an embodiment, the virtual rendering system 210 can switch between tracking systems based on the availability of tracked markers to maintain consistent tracking in accordance with 6DoF.

[0080] The interruption detection component 606 monitors the tracking markers to detect interruptions. When the interruption detection component 606 detects an interruption in one or more markers, such that tracking in 6 degrees of freedom becomes unreliable or impossible, the virtual rendering system 210 switches to tracking the virtual modifications in 3D space in 3 degrees of freedom to prevent interruption of the display. For example, the virtual rendering system 210 can switch from a first tracking system (or a first tracking system set in a set of tracking systems) to a second tracking system (or a second tracking system set in a set of tracking systems), wherein the second tracking system is capable of tracking the virtual modifications in 3D space in 3 degrees of freedom based on the available tracking markers.

[0081] In some example embodiments, the tracking system set of the virtual rendering system 210 includes a gyroscope tracking system, an NFT system, and a SLAM tracking system. Each tracking system in the tracking system set can analyze tracking markers to track the position of a virtual object in 3D space. For example, in order to track a virtual object with 6DoF, the virtual rendering system 210 may require at least six tracking markers to be available. When a tracking marker becomes obstructed or unavailable for various reasons, the virtual rendering system 210 can switch between the available tracking systems in the tracking system set to maintain 6DoF or switch to 3DoF when necessary.

[0082] It will be readily appreciated that the virtual rendering system 210 provides consistent rendering of virtual modifications (e.g., visual effects applied to real-world surfaces) in real-world 3D space in a variety of environments and situations. In many applications, it may be desirable to provide robust consistency in the positions of these virtual modifications as one or more users, cameras, or other tracked items move around the environment. This may involve the identification and use of specific fixed reference points (e.g., fixed surfaces) in the real-world environment. Failure to use fixed reference points or items can result in floating or other undesirable inconsistencies in the rendering and presentation of virtual objects.

[0083] To ensure consistent positioning of virtual objects, annotation data in the form of an example rendering lens, specific to the virtual modification tracking and rendering described herein, may be employed. Specifically, a surface-aware lens is a rendering lens that identifies and references real-world surfaces (e.g., the ground) to facilitate consistent rendering and presentation of virtual modifications in 3D space. The surface-aware lens may be a specific portion or sub-component within the rendering component 602. This surface-aware lens of the rendering component 602 may be configured to identify a reference surface based on visual camera content and may also utilize other device inputs (e.g., a gyroscope, accelerometer, compass) to determine the appropriate surface within the 3D space depicted in the live camera feed. Once the reference surface is determined, virtual modifications may be performed relative to that reference surface. In one example, the reference surface in 3D space is the ground. The virtual rendering system 210 may modify the ground depicted in the live camera feed by applying visual effects to the ground. The virtual rendering system 210 may also render virtual objects at a location in 3D space so that the title appears anchored to the ground.

[0084] In some embodiments, the virtual rendering system 210 can render a virtual modification of the 3D space depicted in the live camera feed of the client device 102 in response to a trigger event. To this end, the event detection component 610 is responsible for detecting such a trigger event. The event detection component 610 can detect the trigger event based on data received from one or more components of the client device 102 or from one or more external sources accessible via the network 106. For example, the trigger event can be based on geolocation data from the location component of the client device 102, and the detection of the trigger event can include detecting that the client device 102 is at or near a specific geographic location. As another example, the trigger event can be based on a time factor, and the detection of the trigger event can include detecting a specific date or time based on a clock signal maintained by the client device 102. As yet another example, the trigger event can be based on weather data describing weather conditions (e.g., obtained from an external source via the network 106), and the detection of the trigger event can include detecting specific weather conditions (e.g., snow, rain, wind, etc.).

[0085] Figure 7 is a flow chart illustrating a method 700 for rendering a virtual modification in a 3D space according to various embodiments of the present disclosure. Method 700 may be embodied in computer-readable instructions that are executed by one or more processors, such that the operations of method 700 may be performed in part or in whole by functional components of virtual rendering system 210; accordingly, method 700 is described below with reference thereto by way of example. However, it should be understood that at least some operations of method 700 may be deployed on various other hardware configurations, and method 700 is not intended to be limited to virtual rendering system 210.

[0086] As depicted in operation 702, the virtual rendering system 210 receives input to activate the surface awareness lens. The input can be in the form of a manual user input, which can be, for example, a button tap or holding or pointing the active camera in a manner indicating selection of the surface awareness lens. The surface awareness lens can be used, for example, with any virtual object for which the object template component 608 maintains a template, although the surface awareness lens is not limited in application to virtual object templates maintained by the object template component 608.

[0087] In operation 704, the rendering component 602 responds to the input by detecting a real-world reference surface in the 3D space depicted in the live camera feed generated by the camera. The camera feed includes image data, including a sequence of images (e.g., a video) depicting the 3D space. In some embodiments, the reference surface can be a user-specified reference surface. Thus, detection of the reference surface is based on user input, such as a tap or other gesture used to activate a surface lens to indicate a reference surface. In many cases, this reference surface can be a floor surface or the ground, but other fixed and identifiable surfaces can also be used. For example, the rendering component 602 can determine the reference surface by identifying a fixed surface based on analysis of visual camera content, and can also utilize other device inputs (e.g., a gyroscope, accelerometer, compass) to determine the appropriate surface within the 3D space captured by the camera feed. In various embodiments, the user can be asked to confirm that a suitable reference surface has been indicated or highlighted. In some cases, the system can indicate that a suitable reference surface cannot be detected, thereby potentially requiring further input or assistance from the user.

[0088] At operation 706, the rendering component 602 orients the virtual modification based on the detected reference surface. Orienting the virtual modification can include assigning the virtual modification (e.g., a virtual object) to a location in 3D space based on the detected reference surface and identifying tracking markers to be used by the tracking system 604 when tracking the virtual object in 3D space. The location to which the virtual modification is assigned can correspond to the reference surface or a predetermined distance above the reference surface. One or both of operations 704 and 706 can also be referred to as initialization of the rendering component 602. Essentially, the determined reference surface within the camera feed is established in the rendering component 602 at an appropriate static orientation relative to the reference surface in the real world.

[0089] In operation 708, the rendering component 602 renders the virtual modification relative to the reference surface. For example, the rendering component 602 may render a virtual object relative to the reference surface. Rendering the virtual object relative to the reference surface may include rendering and maintaining the virtual object at a designated location within 3D space. Thus, in cases where the designated location is a predetermined distance from the reference surface, the rendering of the virtual object may include rendering and maintaining the virtual object at the predetermined distance from the reference surface. In these cases, the virtual object may not actually be in contact with or resting against the reference surface when rendered, but may instead hover above the reference surface or extend a predetermined distance away from the reference surface.

[0090] Figure 8 is a flow chart illustrating the operation of the virtual rendering system 210 in executing a method 800 for tracking virtual modifications at a location relative to the client device 102 in 3D space, according to certain example embodiments. The method 800 may be embodied in computer-readable instructions that are executed by one or more processors, such that the operations of the method 800 may be performed in part or in whole by functional components of the virtual rendering system 210; accordingly, the method 800 is described below with reference thereto by way of example. However, it should be understood that at least some operations of the method 800 may be deployed on various other hardware configurations, and the method 800 is not intended to be limited to the virtual rendering system 210.

[0091] At operation 802 , the rendering component 602 renders in 3D space a virtual modification to the 3D space depicted in the camera feed at a location relative to the client device 102 .

[0092] At operation 804, the tracking system 604 tracks the virtual modification at a location in 3D space in 6 degrees of freedom (6DOF) based on the set of tracking markers via the first tracking subsystem 604A or a combination of multiple tracking subsystems (e.g., the first tracking subsystem 604A and the second tracking subsystem 604B). When tracking the virtual modification in 6DOF, a user viewing the modification on the client device 102 can turn or move in any direction without interrupting tracking of the modification. For example, the tracking system 604 can track the location of the virtual modification based on a combination of an NFT system and a gyroscope tracking system.

[0093] At operation 806, the interruption detection component 606 detects an interruption of a tracking marker from among the tracking markers tracked by a tracking subsystem (e.g., the first tracking subsystem 604A). For example, the first tracking subsystem 604A can include an NFT system configured to rely on a tracking marker comprising a feature of the environment or an active light source proximate to the virtual modification within the environment (e.g., the ground plane or the horizon). The NFT system of the first tracking subsystem 604A can thus rely on the positions of three or more known features in the environment to determine the position of the virtual modification in 3D space relative to the client device 102. If any one or more tracking markers tracked by the first tracking subsystem 604A become obstructed or unavailable, tracking of the virtual modification in 3D space will be interrupted.

[0094] At operation 808, in response to the interruption detection component 606 detecting an interruption of one or more tracking markers, the tracking system 604 switches to one or more other tracking subsystems (e.g., the second tracking subsystem 604B and / or the third tracking subsystem 604C) to maintain tracking of the virtual object in 3D space relative to the client device 102. In doing so, the virtual rendering system 210 can switch from 6DoF to 3DoF, where 3DoF measures pitch, roll, and yaw, but not translation. As the tracking markers become available again, the virtual rendering system 210 can thereby switch back from 3DoF to 6DoF. For example, when the NFT system becomes unavailable, the tracking system 604 can utilize the last tracking markers collected and tracked by the NFT system in subsequent 3DoF experiences.

[0095] Figure 9-13is a flow chart illustrating example operations of a virtual rendering system in executing method 900 for rendering a virtual modification to a surface in 3D space, according to an example embodiment. Method 900 may be embodied in computer-readable instructions that are executed by one or more processors, such that the operations of method 900 may be performed in part or in whole by functional components of virtual rendering system 210; therefore, method 900 is described below with reference thereto by way of example. However, it should be understood that at least some operations of method 900 may be implemented on various other hardware configurations, and method 900 is not intended to be limited to virtual rendering system 210.

[0096] At operation 902, the rendering component 602 detects a reference surface in a 3D space depicted in a camera feed generated by a camera of a computing device (e.g., client device 102). The camera feed contains a sequence of images, each depicting a 3D space. As previously described, the reference surface can be the ground, but any other fixed and determinable surface can also be used. For example, the rendering component 602 can detect the reference surface by identifying a fixed surface based on analysis of visual camera content, and can also utilize other device inputs (e.g., gyroscope, accelerometer, compass) to determine what is a suitable surface within the 3D space depicted in the camera feed.

[0097] In some embodiments, the detection of a reference surface may be based on user input received on the presentation of the camera feed. This input may be in the form of a manual user input, which may be, for example, a button tap or holding or pointing the active camera in a manner that indicates that a surface is being referenced. In other embodiments, this will be referenced below. Figure 10 As discussed, the detection of the reference surface may be in response to detecting a triggering event associated with the reference surface.

[0098] At operation 904, the rendering component 602 dynamically applies an image mask to a reference surface within the 3D space depicted within the camera feed. More specifically, the rendering component 602 applies the image mask to each of the multiple images in the camera feed at the location of the reference surface depicted in each image, which may vary due to changes in the orientation or distance of the camera relative to the reference surface. Typically, applying the image mask to each image includes classifying pixels in the image based on whether they are within or outside the boundaries of the reference surface. This pixel classification process can be based on a determined geometry of the 3D space, a determined color region within the image, or a determined photometric consistency region within the image.

[0099] In some embodiments, application of the image mask can include applying an image segmentation neural network to a plurality of images fed by the camera. The image segmentation neural network can be trained to perform image segmentation on the images to segment each image into a plurality of image segments (e.g., sets of pixels). More specifically, the image segmentation neural network can be trained to assign a first label to pixels within a reference surface boundary and a second label to pixels outside the reference surface boundary. Applying the image segmentation neural network to the images fed by the camera produces a segmented image having two image segments—a first image segment that includes a first set of pixels assigned to a first label corresponding to the reference surface; and a second image segment that includes a second set of pixels assigned to a second label corresponding to the remainder of the 3D space depicted in the image.

[0100] At operation 906, the rendering component 602 applies a visual effect to the image mask corresponding to the reference surface. Applying the visual effect to the image mask causes the modified surface to be rendered in the presentation of the camera feed on the display of the computing device. The visual effect can be any of a variety of visual effects including, for example, changing the color of the surface, changing the texture of the surface, applying an animation effect (e.g., flowing water) to the surface, blurring the surface, rendering a moving virtual object whose movement is constrained by the boundaries of the surface, replacing the surface with other visual content, and various combinations thereof. Figure 14A and 14B An example application of visual effects to a reference surface is shown in and described below.

[0101] like Figure 10 As shown, in some embodiments, method 900 may further include operations 901 and 905. Operation 901 may be performed before operation 902 in which rendering component 602 detects a reference surface in a 3D space captured in a camera feed. In operation 901, event detection component 610 detects a triggering event. For example, a triggering event may be based on geolocation data from a location component of a computing device. As an example, detection of a triggering event may include detecting when the computing device is within a predefined distance of a geographic location. As an additional example, a triggering event may be based on a time factor, and thus, detecting a triggering event may include detecting a specific date or time. As yet another example, a triggering event may be based on weather conditions, and thus, detection of a triggering event may include detecting specific weather conditions (e.g., snow, rain, wind, etc.).

[0102] Operation 905 can be performed before operation 906, in which the rendering component 602 applies a visual effect to the image mask. In operation 905, the rendering component 602 selects a visual effect to be subsequently applied to the image mask. In some embodiments, the rendering component 602 selects a visual effect based on previous user input specifying a particular visual effect.

[0103] In some embodiments, the triggering event detected in operation 901 is associated with a particular visual effect, and thus, the rendering component 602 may select a graphical event based on the triggering event. For example, the particular visual effect may be associated with a particular geographic location, and when the event detection component 610 detects that the computing device is within a predefined distance of the geographic location, the rendering component 602 selects the visual effect associated with the geographic location.

[0104] like Figure 11 As mentioned, in some embodiments, method 900 may further include operations 1102, 1104, and 1106. One or more of operations 1102, 1104, and 1106 may be performed as part (e.g., a predecessor task, a subroutine, or a portion) of operation 904 in which rendering component 602 applies an image mask to a reference surface. Operations 1102, 1104, and 1106 are described below with reference to a single image from a camera feed, but it should be understood that operations 1102, 1104, and 1106 may be repeated for each image in the camera feed to implement dynamic application of the image mask to the reference surface, as depicted throughout the camera feed.

[0105] In operation 1102, the rendering component 602 determines the boundaries of a reference surface within the 3D space depicted in the image fed by the camera. In determining the boundaries of the reference surface, the rendering component 602 may, for example, employ one of many known edge detection techniques. Figure 12 and 13 Further details regarding determining the boundaries of the reference surface are discussed.

[0106] In operation 1104, the rendering component 602 classifies pixels in the image that are within the boundary of the reference surface according to the first class. In operation 1106, the rendering component 602 classifies pixels in the image that are outside the boundary of the reference surface according to the second class. The pixels classified according to the first class form an image mask for the reference surface.

[0107] It should be understood that although Figure 11 The operation of determining the boundary of the reference surface (operation 1102 ) is shown as separate and distinct from operations 1104 and 1106 , but in some embodiments, the determination of the boundary of the reference surface may be combined and performed as part of operations 1104 and 1106 .

[0108] like Figure 12As mentioned above, in some embodiments, method 900 may further include operations 1202, 1204, 1206, 1208, 1210, and 1212. In some embodiments, operations 1202, 1204, 1206, 1208, 1210, and 1212 may be performed as part of (e.g., a predecessor task, subroutine, or portion of) operation 1004 in which rendering component 602 applies an image mask to a reference surface. In some embodiments, operations 1202, 1204, 1206, 1208, 1210, and 1212 may be performed as part of (e.g., a predecessor task, subroutine, or portion of) operation 1202 in which rendering component 602 determines a boundary of a reference surface.

[0109] In operation 1202, the rendering component 602 obtains a set of points from a plurality of images in a camera feed. Obtaining the set of points may include sampling a set of pixels randomly selected from the camera feed.

[0110] At operation 1204, the rendering component 602 associates a first subset of the point set with a first plane in the 3D space depicted in the image. At operations 1206-1208, the rendering component 602 associates a second through Nth subset of the point set with a second through Nth plane in the 3D space depicted in the image, respectively. The rendering component 602 may associate each subset of the point set with a corresponding plane in the 3D space based on the determined geometry of the 3D space, sensor data from one or more sensors (e.g., a gyroscope and accelerometer) of the computing device, location data from one or more location components (e.g., a compass and a global positioning system (GPS)), or various combinations of the two. For example, associating the point set with the corresponding plane may include determining a position and orientation of the camera relative to the plane based in part on the sensor data (e.g., a gyroscope and accelerometer), and mapping pixel positions in the image to positions in the 3D space based in part on the location data.

[0111] In operation 1210, the rendering component 602 determines that the first plane corresponds to the detected reference surface. In operation 1212, the rendering component 602 identifies a boundary of the first plane and in doing so, the rendering component 602 determines a boundary of the reference surface.

[0112] like Figure 13In some embodiments, method 1000 may further include operations 1302, 1304, and 1306. In some embodiments, operations 1302, 1304, and 1306 may be performed as part of (e.g., a predecessor task, subroutine, or portion of) operation 1004 in which rendering component 602 applies the image mask to the reference surface. In some embodiments, operations 1302, 1304, and 1306 may be performed specifically as part of (e.g., a predecessor task, subroutine, or portion of) operation 1202 in which rendering component 602 determines the boundary of the reference surface. Operations 1302, 1304, and 1306 are described below with reference to a single image from a camera feed, but it should be understood that operations 1302, 1304, and 1306 may be repeated for each image in the camera feed to achieve dynamic application of the image mask to the reference surface, as depicted throughout the camera feed.

[0113] In operation 1302, the rendering component 602 identifies regions of similar color in an image fed by a camera. The identification of regions of similar color may include associating groups of pixels in the image based on pixel color values. More specifically, the rendering component 602 may associate pixel groups based on pixels within the group having pixel color values that do not exceed a threshold standard deviation.

[0114] In operation 1304, the rendering component 602 determines which similar color region corresponds to the detected reference surface. In operation 1306, the rendering component 602 identifies a boundary of the similar color region corresponding to the reference surface, thereby determining the boundary of the reference surface.

[0115] Figure 14A and 14B is an interface diagram illustrating aspects of an interface provided by the virtual rendering system 210 according to an example embodiment. Figure 14A , shows a camera feed 1400. The camera feed 1400 can be displayed on a display of a computing device (e.g., client device 102), and although Figure 14A Only a single image is illustrated, but camera feed 1400 may include a sequence of images (e.g., a video) produced by a camera of the computing device. Figure 14A The camera feed 1400 is shown displayed alone and may be presented within or as part of a user interface element displayed among other user interface elements that facilitate functionality that allows a user to interact with the camera feed 1400 in various ways.

[0116] A 3D real-world environment is depicted in camera feed 1400. In particular, the 3D real-world environment depicted in the camera feed includes a sidewalk surface 1402. Virtual rendering system 210 can analyze camera feed 1400, thereby identifying sidewalk surface 1402 using other device inputs such as a gyroscope, accelerometer, and compass, and dynamically apply an image mask to sidewalk surface 1402 according to any of the methods described above. Virtual rendering system 210 can modify sidewalk surface 1402 as presented in camera feed 1400 by applying a visual effect to the image mask corresponding to sidewalk surface 1402. Applying the visual effect to the image mask causes the modified surface to be rendered in the presentation of camera feed 1400.

[0117] For example Figure 14B 14. The modified surface 1404 is illustrated as being rendered within the camera feed 1400. The modified surface 1404 corresponds to the sidewalk surface 1402 with the visual effects applied. In particular, the modified surface 1402 includes an animation that causes the sidewalk surface 1402 to appear in the camera feed 1400 as flowing water rather than sidewalk.

[0118] Figure 15 is a block diagram illustrating an example software architecture 1506, which may be used in conjunction with the various hardware architectures described herein. Figure 15 is a non-limiting example of a software architecture, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. The software architecture 1506 may be implemented in a system such as Figure 16 1600 or similar hardware, for example, Figure 16 The machine 1600 includes a processor 1604, a memory 1614, and input / output (I / O) components 1618. Representative hardware layers 1552 are shown and may represent, for example, Figure 16 1600. A representative hardware layer 1552 includes a processing unit 1554 having associated executable instructions 1504. Executable instructions 1504 represent executable instructions of a software architecture 1506, including implementations of the methods, components, and the like described herein. Hardware layer 1552 also includes memory and / or storage 1556, also having executable instructions 1504. Hardware layer 1552 may also include other hardware 1558.

[0119] exist Figure 15In the example architecture of , software architecture 1506 can be conceptualized as a stack of layers, where each layer provides specific functionality. For example, software architecture 1506 may include layers such as operating system 1502, libraries 1520, applications 1516, framework / middleware 1518, and presentation layer 1514. In operation, applications 1516 and / or other components within a layer may invoke API call 1508 through the software stack and receive response 1512 in response to API call 1508. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or proprietary operating systems may not provide framework / middleware 1518, while other operating systems may provide such a layer. Other software architectures may include additional or different layers.

[0120] The operating system 1502 can manage hardware resources and provide common services. The operating system 1502 may include, for example, a kernel 1522, services 1524, and drivers 1526. The kernel 1522 may act as an abstraction layer between the hardware and other software layers. For example, the kernel 1522 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 1524 may provide other common services to other software layers. Drivers 1526 are responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers 1526 may include display drivers, camera drivers, drivers, flash drivers, serial communication drivers (for example, Universal Serial Bus (USB) drivers), drivers, audio drivers, power management drivers, etc.

[0121] Libraries 1520 provide common infrastructure used by applications 1516 and / or other components and / or layers. The functionality provided by libraries 1520 allows other software components to perform tasks more easily than by directly interfacing with underlying operating system 1502 functionality (e.g., kernel 1522, services 1524, and / or drivers 1526). Libraries 1520 may include system libraries 1544 (e.g., C standard libraries), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, libraries 1520 may include API libraries 1546, such as media libraries (e.g., libraries that support rendering and manipulation of various media formats, such as MPREG4, H.264, MP3, AAC, AMR, JPG, and PNG), graphics libraries (e.g., OpenGL frameworks that can be used to render two-dimensional and 3D graphics content on a display), database libraries (e.g., SQLite that can provide various relational database functions), network libraries (e.g., that can provide web browsing functionality), and the like. The library 1520 may also include a variety of other libraries 1548 to provide numerous other APIs to the applications 1516 and other software components / modules.

[0122] Framework / middleware 1518 (sometimes also referred to as middleware) provides a higher-level common infrastructure that can be used by applications 1516 and / or other software components / modules. For example, framework / middleware 1518 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework / middleware 1518 can provide a wide range of other APIs that can be used by applications 1516 and / or other software components / modules, some of which can be specific to a particular operating system 1502 or platform.

[0123] Applications 1516 include built-in applications 1538 and / or third-party applications 1540. Examples of representative built-in applications 1538 may include, but are not limited to, a contacts application, a browser application, a book reader application, a location application, a media application, a messaging application, and / or a game application. Third-party applications 1540 may include applications that are run by entities other than the vendor of a particular platform using Android TM or IOS TM Software Development Kit (SDK) for developing applications and can be used on platforms such as IOS TM ANDROID TM , WINDOWS Phone or other mobile operating systems. The third party application 1540 can call API calls 1508 provided by the mobile operating system (e.g., operating system 1502) to facilitate the functions described herein.

[0124] Applications 1516 may create user interfaces to interact with system users using built-in operating system functionality (e.g., kernel 1522, services 1524, and / or drivers 1526), libraries 1520, and framework / middleware 1518. Alternatively or additionally, in some systems, interaction with the user may be through a presentation layer, such as presentation layer 1514. In these systems, the application / component "logic" may be separated from the aspects of the application / component that interact with the user.

[0125] Figure 16 is a block diagram illustrating components of a machine 1600 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. Figure 16 A diagrammatic representation of a machine 1600 in the example form of a computer system is shown, in which instructions 1610 (e.g., software, a program, an application, an applet, an app, or other executable code) are used to enable the machine 1600 to perform any one or more of the methodologies discussed herein. Thus, the instructions 1610 may be used to implement the modules or components described herein. The instructions 1610 transform a general-purpose, non-programmed machine 1600 into a specialized machine 1600 that is programmed to perform the functions described and illustrated in the manner described. In alternative embodiments, the machine 1600 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1600 may operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1600 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital computer assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1610, sequentially or otherwise, that specify actions to be taken by the machine 1600. Furthermore, while only a single machine 1600 is shown, the term "machine" should also be taken to include an aggregate of machines that individually or jointly execute instructions 1610 to perform any one or more of the methodologies discussed herein.

[0126] The machine 1600 may include a processor 1604, memory / storage 1606, and I / O components 1618, which may be configured to communicate with each other, for example, via a bus 1602. In an example embodiment, the processor 1604 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1608 that may execute instructions 1610 and a processor 1612. The term "processor" is intended to include a multi-core processor 1604, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions concurrently. Although Figure 16 Multiple processors are shown, but machine 1600 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0127] The memory / storage device 1606 may include a memory 1614, such as a main memory or other memory, and a storage unit 1616, both of which are accessible by the processor 1604, such as via the bus 1602. The storage unit 1616 and the memory 1614 store instructions 1610 that embody any one or more of the methodologies or functionality described herein. The instructions 1610 may also reside, completely or partially, within the memory 1614, within the storage unit 1616, within at least one of the processors 1604 (e.g., within a cache of the processor), or any suitable combination thereof during execution thereof by the machine 1600. Thus, the memory 1614, the storage unit 1616, and the memory of the processor 1604 are examples of machine-readable media.

[0128] The I / O components 1618 may include a variety of components to receive input, provide output, generate output, transmit information, exchange information, capture measurements, and the like. The specific I / O components 1618 included in a particular machine 1600 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that the I / O components 1618 may include Figure 16Many other components are not shown in the figure. To simplify the following discussion, the I / O components 1618 are grouped according to function, and this grouping is by no means limiting. In various example embodiments, the I / O components 1618 may include output components 1626 and input components 1628. The output components 1626 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1628 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide position and / or touch force or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0129] In further example embodiments, the I / O components 1618 may include a biometric component 1630, a motion component 1634, an environmental component 1636, or a positioning component 1638, among a wide range of other components. For example, the biometric component 1630 may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying individuals (e.g., voice recognition, retina recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1634 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1636 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases to ensure safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The positioning component 1638 may include a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), and the like.

[0130] Communication can be accomplished using a variety of technologies. I / O components 1618 may include communication components 1640 operable to couple machine 1600 to network 1632 or device 1620 via coupling 1624 and coupling 1622, respectively. For example, communication components 1640 may include a network interface component or other suitable device for interfacing with network 1632. In further examples, communication components 1640 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, components (e.g., low power ), Components and other communication components to provide communication through other means. Device 1620 can be another machine or various peripheral devices (e.g., peripheral devices coupled via USB).

[0131] In addition, the communication component 1640 can detect an identifier or include components operable to detect an identifier. For example, the communication component 1640 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor that detects a one-dimensional barcode such as a Universal Product Code (UPC) barcode, a multi-dimensional barcode such as a Quick Response (QR) code, an Aztec code, a Data Matrix, a Dataglyph, a MaxiCode, a PDF417, an Ultra Code, a UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying a tagged audio signal). In addition, various information can be derived through the communication component 1640, such as location via Internet Protocol (IP), geolocation via Positioning by signal triangulation, positioning by detecting NFC beacon signals that can indicate a specific location, and more.

[0132] the term

[0133] In this context, "carrier signal" refers to any intangible medium that can store, encode, or carry instructions for execution by a machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions can be transmitted or received over a network using a transmission medium via a network interface device and using any of a number of well-known transmission protocols.

[0134] A "client device" in this context refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device can be, but is not limited to, a mobile phone, desktop computer, laptop computer, PDA, smartphone, tablet computer, ultrabook, netbook, notebook computer, multiprocessor system, microprocessor-based or programmable consumer electronics, game console, set-top box, or other communications device that a user can use to access a network.

[0135] In this context, a "communication network" refers to one or more parts of a network, which can be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a telephone service (POTS) network, a cellular telephone network, a wireless network, The network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless or cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of a variety of types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other standards defined by various standards setting organizations, other long range protocols, or other data transmission technologies.

[0136] In this context, "ephemeral messages" are messages that can be accessed for a limited time. Ephemeral messages can be text, images, videos, and more. The access time for ephemeral messages can be set by the sender. Alternatively, the access time can be set by default or specified by the recipient. Regardless of the setting technique, the message is ephemeral.

[0137] In this context, a “machine-readable medium” refers to a component, device, or other tangible medium that is capable of storing instructions and data, either temporarily or permanently, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage (e.g., erasable programmable read-only memory (EPROM)), and / or any suitable combination thereof. The term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store instructions. The term “machine-readable medium” should also be taken to include any medium or combination of multiple media that can store instructions (e.g., code) for execution by a machine, such that the instructions, when executed by one or more processors of the machine, cause the machine to perform any one or more of the methods described herein. Thus, a “machine-readable medium” refers to a single storage device or device, as well as a “cloud-based” storage system or storage network that includes multiple storage devices or devices. The term “machine-readable medium” does not include the signal itself.

[0138] In this context, a "component" refers to a device, a physical entity, or logic with boundaries defined by function or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularization of specific processing or control functions. Components can be combined with other components through their interfaces to perform machine processes. A component can be a packaged functional hardware unit designed for use with other components, and is part of a program that typically performs a specific function of a related function. A component can constitute a software component (e.g., code contained on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that is capable of performing a specific operation and can be configured or arranged in a specific physical manner. In various example embodiments, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or application portion) to operate as a hardware component that performs certain operations as described herein. The hardware component can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component can include dedicated circuits or logic that are permanently configured to perform a specific operation. The hardware component can be a dedicated processor, such as a field programmable gate array (FPGA) or an ASIC. The hardware components may also include programmable logic or circuits that are temporarily configured by software to perform certain operations. For example, the hardware components may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware components become specific machines (or specific components of a machine) that are uniquely customized to perform the configured functions and are no longer general-purpose processors. It should be understood that the hardware components may be mechanically implemented in a circuit that is dedicated and permanently configured, or in a circuit that is temporarily configured (for example, configured by software) by cost and time considerations. Therefore, the phrase "hardware components" (or "hardware-implemented components") should be understood to include tangible entities, i.e., entities that are physically constructed, permanently configured (for example, hardwired) or temporarily configured (for example, programmed), so as to operate or perform certain operations described herein in a certain manner. In view of the embodiment in which the hardware components are temporarily configured (for example, programmed), each hardware component does not need to be configured or instantiated at any one moment. For example, in the case where the hardware components include a general-purpose processor that is configured as a special-purpose processor by software, the general-purpose processor can be configured to be respectively different special-purpose processors (for example, including different hardware components) at different times. Software accordingly configures a particular processor or processors, for example, to constitute a particular hardware component at one time and to constitute a different hardware component at a different time. Hardware components can provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components can be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, signal transmission between two or more of the communicating hardware components (e.g., through appropriate circuits and buses) can achieve communication. In embodiments where multiple hardware components are configured or instantiated at different times, communication between these hardware components can be achieved, for example, by storing and retrieving information in a memory structure accessible to multiple hardware components. For example, a hardware component can perform an operation and store the output of the operation in a storage device to which it is communicatively coupled. Another hardware component can then access the storage device at a later time to retrieve and process the stored output. The hardware component can also initiate communication with an input or output device and can operate on resources (e.g., a collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by a processor, where the specific one or more processors are examples of hardware. For example, at least some operations of the method can be performed by one or more processors or processor-implemented components. In addition, one or more processors can also operate to support the performance of related operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some operations can be performed by a group of computers (as an example of a machine including a processor), which can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The performance of certain operations can be distributed between processors, not only residing in a single machine, but also deployed on multiple machines. In some example embodiments, the processor or the processor-implemented components can be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processor or the processor-implemented components can be distributed across multiple geographic locations.

[0139] A "processor" in this context refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that operates on data values according to control signals (e.g., "commands," "opcodes," "machine code," etc.) and produces corresponding output signals that are used to operate the machine. For example, a processor can be a central processing unit (CPU), a RISC processor, a CISC processor, a GPU, a DSP, an ASIC, a RFIC), or any combination thereof. A processor can also be a multi-core processor having two or more independent processors (sometimes called "cores") that can execute instructions simultaneously.

[0140] A "timestamp" in this context refers to a sequence of characters or encoded information that identifies when a specific event occurred, such as giving a date and time, sometimes accurate to a fraction of a second.

Claims

1. A system for virtual surface modification, comprising: Memory; and at least one hardware processor coupled to the memory and comprising a virtual rendering system including a set of tracking subsystems, the virtual rendering system causing the system for virtual surface modification to perform operations comprising: receiving a note for a virtual modification to be applied at a specified geographic location; Detecting a triggering event for rendering the virtual modification to the real-world surface of the specified geographic location, detecting the triggering event comprises: Detect weather conditions based on weather data; detecting, based on geographic location data from a computing device, that the computing device is within a predetermined distance of the specified geographic location; In response to detecting the triggering event, identifying the real-world surface in a 3D space captured within a camera feed generated by a camera of the computing device, and selecting a visual effect to apply to the real-world surface based on an association of the visual effect with the specified geographic location and the weather condition; and applying a visual effect to the real-world surface in the 3D space captured within the camera feed, the applying of the visual effect to the real-world surface causing the modified surface to be rendered in a presentation of the camera feed on a display of the computing device; Wherein, applying the visual effect to the real-world surface comprises: tracking, via a first tracking subsystem from the set of tracking subsystems, the visual effect at the real-world surface using tracking markers; and In response to detecting an interruption of the tracking mark, switching from tracking the visual effect via the first tracking subsystem to tracking the visual effect via a second tracking subsystem in the set of tracking subsystems.

2. The system of claim 1 , wherein applying the visual effect to the real-world surface further comprises: Boundaries of the real-world surface in a 3D space captured within the camera feed are determined.

3. The system of claim 2, wherein the camera feed comprises a plurality of images; and; in, The application of the visual effects also includes: classifying pixels of a plurality of images within a boundary of the real-world surface according to a first class; and Pixels of the plurality of images outside of a boundary of the real-world surface are classified according to a second class.

4. The system of claim 3, wherein the plurality of images present a plurality of views of the real-world surface.

5. The system of claim 2, wherein: Determining the boundary of the real-world surface includes: obtaining a set of points from the 3D space captured within the camera feed; associating a first subset of the set of points with a first plane in the 3D space; associating a second subset of the set of points with at least a second plane in the 3D space; and determining, based on sensor data from one or more sensors of the computing device, that the first plane corresponds to the real-world surface; and A boundary of the first plane is identified based on the first subset of the set of points.

6. The system of claim 2, wherein: Determining the boundary of the real-world surface includes: identifying regions of similar color within the 3D space captured within the camera feed; determining that a first similar color region in the 3D space corresponds to the real-world surface; and The boundary of the first similar color region is identified.

7. The system of claim 2, wherein the camera feed comprises a plurality of images; and in, Applying the visual effect includes applying an image segmentation neural network to the plurality of images, wherein applying the image segmentation neural network to the plurality of images generates a plurality of segmented images, each of the plurality of segmented images having a plurality of image segments, the plurality of image segments comprising: a first image segment corresponding to the real-world surface; and A second image segment corresponds to a remaining portion of the 3D space captured within the camera feed.

8. The system of claim 1, wherein the detection of the real-world surface is based on user input identifying the real-world surface.

9. The system of claim 1, wherein: Detecting the triggering event also includes detecting a specific date or time.

10. The system of claim 1, wherein: Applying the visual effect to the real-world surface includes one or more of: changing a texture of the real-world surface, blurring the real-world surface, rendering a moving virtual object within the boundaries of the real-world surface, and replacing the real-world surface with visual content.

11. The system of claim 1, wherein: Applying the visual effect to the real-world surface includes applying an animation effect to the real-world surface.

12. The system of claim 1, wherein: The association with the specified geographic location is based on user input.

13. The system of claim 1, wherein: The operations also include selecting the visual effect based on user input.

14. A method for virtual surface modification, comprising: receiving a note for a virtual modification to be applied at a specified geographic location; Detecting a triggering event for rendering the virtual modification to the real-world surface of the specified geographic location, detecting the triggering event comprises: Detect weather conditions based on weather data; detecting, based on geographic location data from a computing device, that the computing device is within a predetermined distance of the specified geographic location; In response to detecting the triggering event, identifying the real-world surface in a 3D space captured within a camera feed generated by a camera of the computing device, and selecting a visual effect to apply to the real-world surface based on an association of the visual effect with the specified geographic location and the weather condition; and applying a visual effect to the real-world surface in the 3D space captured within the camera feed, the applying of the visual effect to the real-world surface causing a modified surface to be rendered in the camera feed on a display of the computing device; Wherein, applying the visual effect to the real-world surface comprises: tracking, via a first tracking subsystem from a set of tracking subsystems, the visual effect at the real-world surface using tracking markers; and In response to detecting an interruption of the tracking mark, switching from tracking the visual effect via the first tracking subsystem to tracking the visual effect via a second tracking subsystem in the set of tracking subsystems.

15. The method of claim 14, wherein applying the visual effect to the real-world surface further comprises: Boundaries of the real-world surface in a 3D space captured within the camera feed are determined.

16. The method of claim 15, wherein the camera feed comprises a plurality of images; as well as in, The application of the visual effects also includes: classifying pixels of a plurality of images within a boundary of the real-world surface according to a first class; as well as Pixels of the plurality of images outside of a boundary of the real-world surface are classified according to a second class.

17. The method of claim 15, wherein determining a boundary of the real-world surface comprises: obtaining a set of points from the 3D space captured within the camera feed; associating a first subset of the set of points with a first plane in the 3D space; associating a second subset of the set of points with at least a second plane in the 3D space; as well as determining, based on sensor data from one or more sensors of the computing device, that the first plane corresponds to the real-world surface; as well as A boundary of the first plane is identified based on the first subset of the set of points.

18. The method of claim 15, further comprising: detecting a surface modification triggering event based on geolocation data from the computing device, the detecting of the real-world surface being responsive to the detecting of the surface modification triggering event; as well as The visual effect is selected from a plurality of visual effects based on an association of the visual effect with the triggering event.

19. The method of claim 15, wherein: The camera feed includes a plurality of images, wherein the plurality of images present a plurality of views of the real-world surface.

20. A non-transitory machine-readable storage medium comprising a virtual rendering system, the virtual rendering system comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform the following operations, the operations comprising: receiving a note for a virtual modification to be applied at a specified geographic location; Detecting a triggering event for rendering the virtual modification to the real-world surface of the specified geographic location, detecting the triggering event comprises: Detect weather conditions based on weather data; detecting, based on geographic location data from a computing device, that the computing device is within a predetermined distance of the specified geographic location; responsive to detecting the triggering event, identifying the real-world surface in a 3D space captured within a camera feed generated by a camera of the computing device, and selecting a visual effect to apply to the real-world surface based on an association of the visual effect with the specified geographic location and the weather condition; applying a visual effect to the real-world surface in the 3D space captured within the camera feed, the applying of the visual effect to the real-world surface causing a modified surface to be rendered in the camera feed on the computing device; Wherein, applying the visual effect to the real-world surface comprises: tracking, via a first tracking subsystem from a set of tracking subsystems, the visual effect at the real-world surface using tracking markers; and In response to detecting an interruption of the tracking mark, switching from tracking the visual effect via the first tracking subsystem to tracking the visual effect via a second tracking subsystem in the set of tracking subsystems.

Images