Nerve coloring of reflective surfaces

Abstract

The subject technology receives a grid of objects, information related to a viewpoint used to render an image of an object having a reflective surface, and a set of maps. The subject technology generates a rasterized RGB (Red Green Blue) image based on an object grid, viewpoints, and a set of maps. The subject technology generates an output image of the object having the reflective surface using a neural network model based at least in part on the rasterized RGB image and the viewpoint. The subject technology provides for displaying an output image of an object having a reflective surface on a display of a computer client device.

Description

[0001] CLAIM OF PRIORITY

[0002] This patent application claims the benefit of priority of U.S. Application Serial No. 18 / 182,117, filed March 10, 2023, which is incorporated herein in its entirety by this reference. BACKGROUND

[0003] Digital images have become a part of everyday life for an increasing number of people as the use of digital images has increased, the affordability of portable computing devices, the availability of increased capacity of digital storage media, and the increased bandwidth and accessibility of network connections. BRIEF DESCRIPTION OF DRAWINGS

[0004] In the drawings, which are not necessarily drawn to scale, like numerals can describe similar components in different views. To easily identify the discussion of any particular element or act, the most significant digit or digits in any

[0005] Figure 1 is a diagrammatic representation of a networking environment in which the present disclosure can be deployed according to some examples.

[0006] Figure 2 is a diagrammatic representation of a messaging system having both client-side functionality and server-side functionality according to some examples.

[0007] Figure 3 is a diagrammatic representation of a data structure as maintained in a database according to some examples.

[0008] Figure 4 is a diagrammatic representation of a message according to some examples.

[0009] Figure 5 An example processing flow for neural shading of a reflective surface is shown according to some implementations of the subject technology.

[0010] Figure 6 Example components used by the processing flow described in Figure 5 are shown according to implementations of the subject technology.

[0011] Figure 7 Examples of irradiance maps and specular maps are shown according to implementations of the subject technology.

[0012] Figure 8 Examples of neural rendering and neural rendering with glossy (e.g., reflective) surface techniques are shown according to some implementations of the subject technology.

[0013] Figure 9 FIG. 1 is a flow diagram illustrating a method in accordance with certain example embodiments.

[0014] Figure 10 FIG. 1 is a flow diagram illustrating a method in accordance with certain example embodiments.

[0015] Figure 11 FIG. 1 is a flow diagram illustrating a method in accordance with certain example embodiments. DETAILED DESCRIPTION

[0016] Users with various interests from various locations can capture digital images of various subjects and make the captured images available to others via a network, such as the Internet. To enhance the user's experience with digital images and provide various features, it can be challenging and computationally intensive to enable a computing device to perform image processing operations on various objects and / or features captured under various varying conditions (e.g., variations in image scale, noise, illumination, motion, or geometric distortions).

[0017] Augmented reality (AR) technology aims to bridge the gap between virtual and real-world environments by providing an augmented real-world environment that is enhanced with electronic information. Thus, the electronic information appears to be part of the real-world environment as perceived by the user. In examples, augmented reality technology also provides a user interface to interact with the electronic information overlaid in the augmented real-world environment.

[0018] As mentioned above, digital images have become a part of everyday life for an increasing number of people as the use of digital images increases, portable computing devices become affordable, digital storage media increase in capacity, and network connections increase in bandwidth and accessibility. Users with various interests from various locations can capture digital images of various subjects and make the captured images available to others via a network, such as the Internet. To enhance the user's experience with digital images and provide various features, it can be challenging and computationally intensive to enable a computing device to perform image processing operations on various objects and / or features captured under various varying conditions (e.g., variations in image scale, noise, illumination, motion, or geometric distortions).

[0019] Implementations of the subject technology can display items (e.g., 3D objects, etc.) in AR applications using techniques described further herein. In a first example, the subject technology creates explicit 3D assets and renders them using physics-based rendering. In a second example, the subject technology displays such assets using neural rendering. In neural rendering, a learned texture is used to perform rasterization on an extracted product mesh (3D geometry), and an image-to-image neural network (trained) outputs a photo-realistic appearance of the product at a particular viewpoint.

[0020] In some existing approaches, neural rendering renders assets with diffuse materials well, but fails to produce reasonable rendering of reflective (shiny) surfaces. In particular, the modeling of specular reflections is poor and results in unrealistic representations of AR objects having such reflective surfaces. Using techniques described further herein, implementations of the subject technology enable neural rendering of shiny surfaces to create more realistic representations of AR objects.

[0021] Messaging systems are frequently used and increasingly used by users of mobile computing devices in various environments to provide different types of functionality in a convenient manner. As described herein, the subject messaging systems include practical applications that provide improvements in capturing image data and rendering AR content (e.g., images, videos, etc.) based on captured image data by providing technical improvements at least in capturing image data using power and resource constrained electronic devices. Such improvements in capturing image data are achieved by techniques provided by the subject technology that reduce latency and improve efficiency in processing captured image data, thereby also reducing power consumption of the capturing device.

[0022] As referred to herein, the phrases“augmented reality experience,”“augmented reality content item,”“augmented reality content generator” include or refer to various image processing operations corresponding to image modification, filtering, AR content generator, media overlay, transformation, etc., as described further herein, and can also include playback of audio or music content during presentation of AR content or media content.

[0023] Networked computing environment

[0024] Figure 1is a block diagram illustrating an example interaction system 100 for facilitating interactions (e.g., exchanging text messages, making text, audio, and video calls, or playing games) on a network. The interaction system 100 includes a plurality of client systems 102, each of which hosts a plurality of applications including an interaction client 104 and other applications 106. Each interaction client 104 is communicatively coupled to other instances of the interaction client 104 (e.g., hosted on respective other user systems 102), an interaction server system 110, and third-party servers 112 via one or more communication networks including a network 108 (e.g., the Internet). The interaction client 104 can also communicate with locally-hosted applications 106 using an application program interface (API).

[0025] Each user system 102 can include a plurality of user devices, such as a mobile device 114, a headset 116, and a computer client device 118, which are communicably connected to exchange data and messages.

[0026] The interaction client 104 interacts with other interaction clients 104 and with the interaction server system 110 via the network 108. Data exchanged between interaction clients 104 (e.g., interactions 120) and between interaction clients 104 and the interaction server system 110 includes functionality (e.g., commands for activating functionality) and payload data (e.g., text, audio, video, or other multimedia data).

[0027] The interaction server system 110 provides server-side functionality to the interaction clients 104 via the network 108. While certain functionality of the interaction system 100 is described herein as being performed by the interaction client 104 or by the interaction server system 110, the location of certain functionality within the interaction client 104 or within the interaction server system 110 can be a design choice. For example, it can be technically preferable to initially deploy a particular technology or functionality within the interaction server system 110, but later migrate that technology or functionality to the interaction client 104 of a user system 102 that has sufficient processing power.

[0028] The interaction server system 110 supports various services and operations provided to the interaction clients 104. Such operations include sending data to the interaction clients 104, receiving data from the interaction clients 104, and processing data generated by the interaction clients 104. The data can include message content, client device information, geolocation information, media augmentations and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the interaction system 100 is activated and controlled through functionality available via a user interface (UI) of the interaction client 104.

[0029] Now turning specifically to the interaction server system 110, an application program interface (API) server 122 is coupled to and provides a programmatic interface to an interaction server 124 that exposes the functions of the interaction server 124 to the interaction clients 104, other applications 106, and third-party servers 112. The interaction server 124 is communicatively coupled to a database server 126 that facilitates access to a database 128 that stores data associated with interactions processed by the interaction server 124. Similarly, a web server 130 is coupled to the interaction server 124 and provides a web-based interface to the interaction server 124. To this end, the web server 130 processes incoming network requests through the Hypertext Transfer Protocol (HTTP) and several related protocols.

[0030] The application program interface (API) server 122 receives and transmits interaction data (e.g., commands and message payloads) between the interaction server 124 and the client systems 102 (and, e.g., the interaction clients 104 and other applications 106) and the third-party servers 112. Specifically, the application program interface (API) server 122 provides a set of interfaces (e.g., routines and protocols) that the interaction clients 104 and other applications 106 can call or query to activate the functions of the interaction server 124. The application program interface (API) server 122 exposes various functions supported by the interaction server 124, including account registration; login functionality; sending interaction data from a particular interaction client 104 to another interaction client 104 via the interaction server 124; transferring media files (e.g., images or videos) from the interaction client 104 to the interaction server 124; setting a collection of media data (e.g., a story); retrieving a list of friends of a user of a user system 102; retrieving messages and content; adding and deleting entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends within a social graph; and opening an application event (e.g., related to an interaction client 104).

[0031] The interaction server 124 hosts a number of systems and subsystems, described below with respect to Figure 2

[0032] Linked applications

[0033] ​Returning to the interaction client 104, features and functionality of the external resource (e.g., a linked application 106 or applet) are made available to the user via the interface of the interaction client 104. In this context, "external" refers to the fact that the application 106 or applet is external to the interaction client 104. The external resource is typically provided by a third party, but can also be provided by the creator or provider of the interaction client 104. The interaction client 104 receives a user selection of an option to launch or access features of such an external resource. The external resource can be an application 106 installed on the user system 102 (e.g., a "native app"), or a scaled-down version of an application (e.g., an "applet") hosted on the user system 102 or located remote from the user system 102 (e.g., on a third-party server 112). The scaled-down version of the application includes a subset of the features and functionality of the application (e.g., a full-scale native version of the application), and is implemented using a markup language document. In some examples, the scaled-down version of the application (e.g., an "applet") is a web-based markup language version of the application, and is embedded in the interaction client 104. In addition to using a markup language document (e.g., a.html file), an applet can include a scripting language (e.g., a.js file or a.json file) and a style sheet (e.g., a.ss file).

[0034] In response to receiving a user selection of an option to launch or access features of an external resource, the interaction client 104 determines whether the selected external resource is a web-based external resource or a locally installed application 106. In some cases, an application 106 that is locally installed on the user system 102 can be launched independently of and separately from the interaction client 104, e.g., by selecting an icon corresponding to the application 106 on a home screen of the user system 102. A scaled-down version of such an application can be launched or accessed via the interaction client 104, and in some examples, no part of the scaled-down application can be accessed outside of the interaction client 104 or a limited portion of the scaled-down application can be accessed outside of the interaction client 104. A scaled-down application can be launched by the interaction client 104 receiving a markup language document associated with the scaled-down application from a third-party server 112 and processing such a document.

[0035] In response to determining that the external resource is a locally installed application 106, the interaction client 104 instructs the user system 102 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interaction client 104 communicates with the third-party server 112, e.g., to obtain a markup language document corresponding to the selected external resource. The interaction client 104 then processes the obtained markup language document to present the web-based external resource within the user interface of the interaction client 104.

[0036] The interaction client 104 can notify users of the user system 102 or other users (e.g., “friends”) related to such users of activity occurring in one or more external resources. For example, the interaction client 104 can provide participants in a conversation (e.g., chat session) in the interaction client 104 with notifications related to the current or recent use of external resources by one or more members of a user group. One or more users can be invited to join an active external resource or launch a recently used but not currently active external resource (in the friend group). The external resource can provide participants in the conversation who are each using a respective interaction client 104 with the ability to share an item, condition, state, or location in the external resource with one or more members of the user group in the chat session. The shared item can be an interactive chat card that members of the chat can utilize to interact, such as to launch the corresponding external resource, view particular information within the external resource, or bring the members of the chat to a particular location or state within the external resource. Within a given external resource, response messages can be sent to users on the interaction client 104. Based on the current context of the external resource, the external resource can selectively include different media items in the response.

[0037] The interaction client 104 can present a list of available external resources (e.g., applications 106 or widgets) to a user to launch or access a given external resource. The list can be presented in the manner of a contextually relevant menu. For example, icons representing different applications (or widgets) of the applications 106 (or widgets) can vary based on how the user launches the menu (e.g., from a conversation interface or from a non-conversation interface).

[0038] System Architecture

[0039] Figure 2 is a block diagram illustrating additional details regarding the interaction system 100 according to some examples. In particular, the interaction system 100 is shown to include the interaction client 104 and the interaction server 124. The interaction system 100 includes a number of subsystems that are supported on the client side by the interaction client 104 and on the server side by the interaction server 124. Example subsystems are discussed below.

[0040] The image processing system 202 provides various functionality that enables a user to capture and enhance (e.g., annotate or otherwise modify or edit) media content associated with a message.

[0041] In some example implementations, the image processing system 202 provides a graphics processing pipeline architecture that enables different augmented reality experiences (e.g., AR content generators) to be applied in corresponding different layers. Such a graphics processing pipeline provides a scalable rendering engine for providing multiple augmented reality experiences that are included in a composite media (e.g., an image or video) or composite AR content for rendering by the interactive client 104.

[0042] The image processing system 202 performs rendering of AR content based on data provided by at least one of the foregoing modules for display by the interactive client 104. In an example, the image processing system 202 uses a graphics processing pipeline to perform graphics operations to render AR content for display. In an example, the image processing system 202 implements a scalable rendering engine that supports multiple image processing operations corresponding to individual augmented reality content generators. In an example, the image processing system 202 can receive composite AR content for rendering on a display provided by the client device 102.

[0043] In some implementations, the image processing system 202 provides a graphics system that renders two-dimensional (2D) objects or (real or imagined) objects from a three-dimensional (3D) world onto a 2D display screen. Such a graphics system (e.g., included on the client device 102) includes, in some implementations, a graphics processing unit (GPU) for performing image processing operations and rendering graphics elements for display.

[0044] In implementations, a GPU includes a logical graphics processing pipeline that can receive a representation of a 2D or 3D scene and provide an output of a bitmap representing a 2D image for display. Existing application program interfaces (APIs) have implemented graphics pipeline models. Examples of such APIs include the OPEN GL API and the METAL API. A graphics processing pipeline includes multiple stages that convert a set of vertices, textures, buffers, and state information into an image frame on a screen.

[0045] In implementations, one of the stages of the graphics processing pipeline is a shader, which can be used as part of a particular augmented reality content generator that is applied to an input frame (e.g., an image or video). Shaders can be implemented as code that runs on a specialized processing unit (also referred to as a shader unit or shader processor) that typically executes several compute threads, which are programmed to generate an appropriate level of color and / or special effects for a fragment being rendered. For example, vertex shaders process attributes of vertices (position, texture coordinates, color, etc.), and pixel shaders process attributes of pixels (texture values, color, z-depth, and alpha values). In some cases, pixel shaders are referred to as fragment shaders.

[0046] It should be appreciated that other types of shader processing can be provided. In examples, an entire frame is rendered using a particular sampling rate within the graphics processing pipeline, and / or pixel shading is performed at a particular rate per pixel. In this way, a given electronic device (e.g., client device 102) operates the graphics processing pipeline to convert information corresponding to an object into a bitmap that can be displayed by the electronic device.

[0047] The camera system 204 includes control software (e.g., in a camera application) that interacts with and controls hardware of the hardware camera of the user system 102 (e.g., directly or via operating system controls) to modify and enhance live images captured and displayed via the interactive client 104.

[0048] The augmentation system 206 provides functionality related to the generation and publication of augmentations (e.g., media overlays) for images captured in real-time by the camera of the user system 102 or images retrieved from memory of the user system 102. For example, the augmentation system 206 is operable to select, present, and display media overlays (e.g., image filters or image lenses) for the interactive client 104 for augmenting live images received via the camera system 204 or stored images retrieved from memory of the user system 102. These augmentations are selected by the augmentation system 206 based on some inputs and data, such as:

[0049] • a geographic location of the user system 102; and

[0050] • social network information of a user of the user system 102.

[0051] The augmentations can include audio and visual content as well as visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. The audio and visual content or visual effects can be applied to media content items (e.g., photos or videos) at the user system 102 for communication in a message or to video content, such as a video content stream or feed sent from the interactive client 104. Thus, the image processing system 202 can interact with and support various subsystems of the communication system 208, such as the messaging system 210 and the video communication system 212.

[0052] The media overlays can include text or image data that can be overlaid on a photo taken by the user system 102 or a video stream made by the user system 102. In some examples, the media overlays can be location overlays (e.g., Venice Beach), name of a live event, or a business name overlay (e.g., Beachside Café). In further examples, the image processing system 202 uses a geographic location of the user system 102 to identify a media overlay that includes a name of a business at the geographic location of the user system 102. The media overlays can include other indicia associated with the business. The media overlays can be stored in the database 128 and accessed by the database server 126.

[0053] The image processing system 202 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify circumstances under which a particular media overlay should be provided to other users. The image processing system 202 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0054] The augmentation creation system 214 supports an augmented reality developer platform and includes an application for content creators (e.g., artists and developers) to create and publish augmentations (e.g., augmented reality experiences) for the interactive client 104. The augmentation creation system 214 provides a library of built-in features and tools for content creators, including, for example, custom shaders, tracking techniques, and templates.

[0055] In some examples, the augmentation creation system 214 provides a business-based publishing platform that enables businesses to select particular augmentations associated with geographic locations via a bidding process. For example, the augmentation creation system 214 associates a media overlay of a highest bidding business with a corresponding geographic location for a predefined amount of time.

[0056] The communication system 208 is responsible for enabling and handling various forms of communication and interaction within the interaction system 100, and includes a messaging system 210, an audio communication system 216, and a video communication system 212. The messaging system 210 is responsible for enforcing temporary or time-limited access to content by the interaction clients 104. The messaging system 210 includes a plurality of timers (e.g., in a transient timer system 218) that selectively enable access (e.g., for presentation and display) of messages and associated content via the interaction clients 104 based on a duration and display parameters associated with a message or collection of messages (e.g., a story). Additional details regarding the operation of the transient timer system 218 are provided below. The audio communication system 216 enables and supports audio communication (e.g., real-time audio chat) between a plurality of interaction clients 104. Similarly, the video communication system 212 enables and supports video communication (e.g., real-time video chat) between a plurality of interaction clients 104.

[0057] The user management system 220 is operationally responsible for managing user data and profiles, and includes a social network system 222 that maintains information about relationships between users of the interaction system 100.

[0058] The collection management system 224 is operationally responsible for managing collections or sets of media (e.g., collections of text, image, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event libraries" or "event stories." Such collections can be made available for a specified period of time (e.g., the duration of an event to which the content relates). For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 224 can also be responsible for publishing icons to the user interfaces of the interaction clients 104 that provide notifications of particular collections. The collection management system 224 includes curation functionality that enables a curator to manage and curate particular collections of content. For example, a curation interface enables an event organizer to curate a collection of content related to a particular event (e.g., to delete inappropriate content or redundant messages). Additionally, the collection management system 224 employs machine vision (or image recognition technology) and content rules to automatically curate collections of content. In certain examples, users can be paid compensation for including user-generated content into a collection. In such cases, the collection management system 224 operates to automatically make payments to such users for use of their content.

[0059] The map system 226 provides various geographic location functionality and supports the presentation of map-based media content and messages by the interactive clients 104. For example, the map system 226 enables the display of user icons or avatars (e.g., stored in the profile data 302) on a map to indicate the current or past locations of a user's "friends" within the context of the map, as well as media content (e.g., collections of messages including photos and videos) generated by such friends. For example, a message posted by a user to the interactive system 100 from a particular geographic location can be displayed to a particular user's "friends" within the context of that particular location on a map interface of the interactive client 104. A user can also share his or her location and status information with other users of the interactive system 100 via the interactive client 104 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to selected users within the context of a map interface of the interactive client 104.

[0060] The game system 228 provides various game functionality within the context of the interactive clients 104. The interactive clients 104 provide a game interface that provides a list of available games that can be launched by a user within the context of the interactive client 104 and played with other users of the interactive system 100. The interactive system 100 also enables a particular user to invite other users to participate in playing a particular game by issuing invitations from the interactive client 104 to the other users. The interactive clients 104 also support voice, video, and text messaging (e.g., chat) within the context of playing games, provide leaderboards for games, and also support the provision of in-game rewards (e.g., game currency and items).

[0061] The external resource system 230 provides an interface for the interactive clients 104 to communicate with remote servers (e.g., third-party servers 112) to launch or access external resources (i.e., applications or applets). Each third-party server 112 hosts, for example, a markup language (e.g., HTML5) based application or a scaled down version of an application (e.g., a game application, a utility application, a payment application, or a ride-sharing application). The interactive clients 104 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from a third-party server 112 associated with the web-based resource. The applications hosted by the third-party servers 112 are programmed in JavaScript with a software development kit (SDK) provided by the interactive server 124. The SDK includes an application program interface (API) with functionality that can be called or activated by the web-based application. The interactive server 124 hosts a JavaScript library that provides given external resource access to particular user data of the interactive clients 104. HTML5 is an example of a technology for programming games, but applications and resources programmed based on other technologies can be used.

[0062] To integrate the functionality of the SDK into a web-based resource, the SDK is downloaded from the interaction server 124 by the third-party server 112, or otherwise received by the third-party server 112. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functionality of the SDK to integrate the features of the interaction client 104 into the web-based resource.

[0063] The SDK stored on the interaction server system 110 effectively provides a bridge between the external resource (e.g., application 106 or widget) and the interaction client 104. This gives the user a seamless experience of communicating with other users on the interaction client 104 while also preserving the look and feel of the interaction client 104. To bridge the communication between the external resource and the interaction client 104, the SDK facilitates the communication between the third-party server 112 and the interaction client 104. The WebView JavaScript Bridge running on the user system 102 establishes two one-way communication channels between the external resource and the interaction client 104. Messages are sent asynchronously between the external resource and the interaction client 104 via these communication channels. Each SDK function activation is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with the callback identifier.

[0064] By using the SDK, not all information from the interaction client 104 is shared with the third-party server 112. The SDK limits which information is shared based on the needs of the external resource. Each third-party server 112 provides the interaction server 124 with an HTML5 file corresponding to the web-based external resource. The interaction server 124 can add a visual representation (e.g., a box design or other graphic) of the web-based external resource in the interaction client 104. Once the user selects the visual representation or indicates through the GUI of the interaction client 104 that the interaction client 104 is to access the features of the web-based external resource, the interaction client 104 obtains the HTML5 file and instantiates the resource for accessing the features of the web-based external resource.

[0065] The interaction client 104 presents a graphical user interface (e.g., a landing page or title screen) for the external resource. During, before, or after presenting the landing page or title screen, the interaction client 104 determines whether the launched external resource has previously been authorized to access user data of the interaction client 104. In response to determining that the launched external resource has previously been authorized to access user data of the interaction client 104, the interaction client 104 presents another graphical user interface of the external resource that includes functionality and features of the external resource. In response to determining that the launched external resource has not previously been authorized to access user data of the interaction client 104, after displaying the landing page or title screen of the external resource for a display threshold period of time (e.g., 3 seconds), the interaction client 104 slides up a menu (e.g., animates the menu to emerge from the bottom of the screen to the middle or other portion of the screen) for authorizing the external resource to access user data. The menu identifies types of user data that the external resource will be authorized to use. In response to receiving a user selection of an accept option, the interaction client 104 adds the external resource to a list of authorized external resources and allows the external resource to access user data from the interaction client 104. The external resource is authorized by the interaction client 104 to access user data under an OAuth 2 framework.

[0066] The interaction client 104 controls the types of user data shared with external resources based on the types of external resources that are authorized. For example, external resources that include full-scale applications (e.g., the application 106) are provided access to a first type of user data (e.g., two-dimensional avatars of users with or without different avatar characteristics). As another example, external resources that include small-scale versions of applications (e.g., web-based versions of the application) are provided access to a second type of user data (e.g., payment information, two-dimensional avatars of users, three-dimensional avatars of users, and avatars with various avatar characteristics). Avatar characteristics include different ways of customizing the appearance of an avatar (e.g., different poses, facial features, clothing, etc.).

[0067] The advertising system 232 is operable to enable third parties to purchase advertisements for presentation to end users via the interaction client 104, and also handles the delivery and presentation of these advertisements.

[0068] Data Architecture

[0069] Figure 3 FIG. 3 is a schematic diagram illustrating a data structure 300 that can be stored in the database 304 of the interaction server system 110, in accordance with certain examples. While the contents of the database 304 are illustrated as including a plurality of tables, it should be understood that data can be stored in other types of data structures (e.g., object-oriented databases).

[0070] The database 304 includes message data stored within a message table 306. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and a payload. Reference is made below to Figure 3 Further details are described regarding information that can be included in messages and included within message data stored in the message table 306.

[0071] An entity table 308 stores entity data and is linked (e.g., by reference) to an entity graph 310 and the profile data 302. Entities for which records are maintained within the entity table 308 can include individuals, corporate entities, organizations, objects, locations, events, and the like. Regardless of entity type, any entity for which the interaction server system 110 stores data can be an identified entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0072] The entity graph 310 stores information about relationships and associations between entities. Such relationships can be social, professional (e.g., working at a common company or organization), interest-based, or activity-based, merely as examples. Certain relationships between entities can be one-way, such as a personal user's subscription to a commercial or publication user's (e.g., a newspaper or other digital media channel or brand) digital content. Other relationships can be two-way, such as a "friend" relationship between individual users of the interaction system 100.

[0073] Certain permissions and relationships can be attached to each relationship, and can also be attached to each direction of a relationship. For example, a two-way relationship (e.g., a friend relationship between individual users) can include authorization for publication of digital content items between the individual users, but can impose certain restrictions or filters (e.g., based on content characteristics, location data, or time-of-day data) on publication of such digital content items. Similarly, a subscription relationship between an individual user and a commercial user can impose varying degrees of restriction on publication of digital content from the commercial user to the individual user, and can significantly restrict or prevent publication of digital content from the individual user to the commercial user. As an example of an entity, a particular user can record certain restrictions in the record for that entity within the entity table 308 (e.g., through privacy settings). Such privacy settings can apply to all types of relationships in the context of the interaction system 100, or can selectively apply to certain types of relationships.

[0074] The profile data 302 stores various types of profile data about a particular entity. Based on privacy settings specified by the particular entity, the profile data 302 can be selectively used and presented to other users of the interaction system 100. In the case of an entity being a person, the profile data 302 includes, for example, the user's name, phone number, address, settings (e.g., notification and privacy settings), and a user-selected avatar representation (or a collection of such avatar representations). The particular user can then selectively include one or more of these avatar representations within the content of messages communicated via the interaction system 100 and on the map interface displayed by the interaction client 104 to other users. The collection of avatar representations can include a "status avatar" that presents a graphical representation of a status or activity that the user can select to communicate at a particular time.

[0075] In the case of an entity being a group, the profile data 302 for the group can similarly include one or more avatar representations associated with the group in addition to the group name, members, and various settings (e.g., notifications) for the relevant group.

[0076] The database 304 also stores augmentation data, such as overlays or filters, in an augmentation table 312. The augmentation data is associated with and applied to videos (data for which is stored in a video table 314) and images (data for which is stored in an image table 316).

[0077] In some examples, a filter is an overlay that is displayed as an overlay on an image or video during presentation to a recipient user. The filter can be of various types, including a user-selected filter from a set of filters presented by the interaction client 104 to a sending user when the sending user is composing a message. Other types of filters include a geo-location filter (also referred to as a geo-filter) that can be presented to the sending user based on a geo-location. For example, a geo-location filter specific to a nearby or special location can be presented by the interaction client 104 within a user interface based on geo-location information determined by a global positioning system (GPS) unit of the user system 102.

[0078] Another type of filter is a data filter that can be selectively presented to the sending user by the interaction client 104 based on other inputs or information collected by the user system 102 during the message creation process. Examples of data filters include a current temperature at a particular location, a current speed at which the sending user is traveling, a battery life of the user system 102, or a current time.

[0079] Other augmentation data that can be stored within the image table 316 includes augmented reality content items (e.g., corresponding to an application "lens" or augmented reality experience). The augmented reality content items can be real-time special effects and sounds that can be added to an image or video.

[0080] Story table 318 stores data regarding collections of messages and associated image, video, or audio data that are compiled into collections (e.g., stories or galleries). The creation of a particular collection can be initiated by a particular user (e.g., each user for which a record is maintained in entity table 308). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. To this end, the user interface of interactive client 104 can include a user-selectable icon to enable a sending user to add particular content to his or her personal story.

[0081] A collection can also constitute a "live story," which is a collection of content from multiple users created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" can constitute a curated stream of user-submitted content from various locations and events. Users whose client devices have location services enabled and are at a common location event at a particular time can be presented with an option to contribute content to a particular live story, e.g., via the user interface of interactive client 104. A live story can be identified to a user by interactive client 104 based on the user's location. The end result is a "live story" told from a group perspective.

[0082] Another type of collection of content is referred to as a "location story," which enables users whose user systems 102 are located within a particular geographic location (e.g., at a college or university campus) to contribute to a particular collection. In some examples, contributions to a location story can employ secondary authentication to verify that an end user belongs to a particular organization or other entity (e.g., is a student in a university campus).

[0083] As mentioned above, video table 314 stores video data that, in some examples, is associated with messages for which records are maintained within message table 306. Similarly, image table 316 stores image data that is associated with messages whose message data is stored in entity table 308. Entity table 308 can associate various augmentations from augmentation table 312 with various images and videos stored in image table 316 and video table 314.

[0084] Data communication architecture

[0085] Figure 4is a schematic diagram showing the structure of a message 400 generated by an interaction client 104 to be transmitted to a further interaction client 104 via an interaction server 124, in accordance with some examples. The content of a particular message 400 is used to populate a message table 306 stored within a database 304 accessible by the interaction server 124. Similarly, the content of a message 400 is stored in memory as “in-transit” or “in-flight” data for the user system 102 or the interaction server 124. The message 400 is shown to include the following example components:

[0086] • message identifier 402: a unique identifier that identifies the message 400.

[0087] • message text payload 404: text to be generated by a user via a user interface of the user system 102 and included in the message 400.

[0088] • message image payload 406: image data captured by a camera component of the user system 102 or retrieved from a memory component in the user system 102 and included in the message 400. Image data for a sent or received message 400 can be stored in an image table 316.

[0089] • message video payload 408: video data captured by a camera component or retrieved from a memory component of the user system 102 and included in the message 400. Video data for a sent or received message 400 can be stored in the image table 316.

[0090] • message audio payload 410: audio data captured by a microphone or retrieved from a memory component of the user system 102 and included in the message 400.

[0091] • message augmentation data 412: augmentation data (e.g., filters, stickers, or other annotations or augmentations) representing an augmentation to be applied to the message image payload 406, message video payload 408, or message audio payload 410 of the message 400. Augmentation data for a sent or received message 400 can be stored in an augmentation table 312.

[0092] • message duration parameter 414: a parameter value indicating, in seconds, an amount of time for the content (e.g., message image payload 406, message video payload 408, message audio payload 410) of the message to be presented to or made accessible to the user via the interaction client 104.

[0093] • Message geolocation parameters 416: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter 416 values can be included in the payload, each of which is associated with a content item included in the content (e.g., a particular image within the message image payload 406 or a particular video in the message video payload 408).

[0094] • Message story identifiers 418: Identifier values that identify one or more content collections (e.g.,“stories” identified in the story table 318) that are associated with particular content items in the message image payload 406 of the message 400. For example, multiple images within the message image payload 406 can each be associated with multiple content collections using identifier values.

[0095] • Message tags 420: Each message 400 can be tagged with multiple tags, each of which indicates a subject matter of content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value can be included within the message tags 420 that indicates the relevant animal. Tag values can be generated manually based on user input, or can be generated automatically using, for example, image recognition.

[0096] • Message sender identifier 422: An identifier (e.g., a messaging system identifier, email address, or device identifier) that indicates a user of a user system 102 on which the message 400 was generated and from which the message 400 was sent.

[0097] • Message recipient identifier 424: An identifier (e.g., a messaging system identifier, email address, or device identifier) that indicates a user of a user system 102 to which the message 400 is addressed.

[0098] The content (e.g., values) of the various components of the message 400 can be pointers to locations in tables where the content data values are stored. For example, the image values in the message image payload 406 can be pointers (or addresses) to locations within the image table 316. Similarly, the values within the message video payload 408 can point to data stored within the image table 316, the values stored within the message augmentation data 412 can point to data stored in the augmentation table 312, the values stored within the message story identifiers 418 can point to data stored in the story table 318, and the values stored within the message sender identifier 422 and the message recipient identifier 424 can point to user records stored within the entity table 308.

[0099] Figure 5An example process flow for a computing architecture for neural shading of reflective surfaces in accordance with some implementations of the subject technology is shown. In implementations, Figure 5 The process flow in FIG. 6 can be performed by the computer client device 118. Alternatively (or in addition), the process flow in FIG. 6 can be performed by the interaction server system 110 or a component thereof (e.g., the interaction server 124). For purposes of discussion, Figure 5 The process flow in FIG. 6 is described with respect to the computer client device 118. In addition, the discussion of Figure 5 The process flow in FIG. 6 is described with respect to the computer client device 118. In addition, the discussion of Figure 5 The discussion of FIG. 6 is discussed in conjunction with portions of Figure 6 The discussion of FIG. 6 is discussed in conjunction with portions of

[0100] In implementations, the training data 510 includes a large number (e.g., 200 or more) of overlapping images of an object from different viewpoints that show the asset uniformly from all sides and are captured in the same environment, with any light represented as an environment map (which is initially unknown). In examples, such overlapping images are captured using a camera of the computer client device 118. The environment map corresponds to a reflection texture that represents a rendering of a scene as seen from a particular position, and a graphics rendering engine of the computer client device 118 can use such a reflection texture to generate reflections on surfaces with metallic materials (or other materials with reflective surfaces).

[0101] In a pre-training phase, the computer client device 118 uses a photogrammetry application (e.g., provided by an internal engine, an API, or a software library, etc.) to identify camera poses for all images (e.g., a set of good light images of an object captured from a number of different angles or viewpoints) and extracts an object mesh 522 (e.g., corresponding to the object in FIG. 5) using the photogrammetry application. As referred to herein, an object mesh is a graphical representation of an object captured from the images referred to above. In implementations, the object mesh includes a set of vertices corresponding to a shape of the object. In implementations, the object mesh can include a set of arrays of triangles, can include information about a set of edges associated with the set of vertices, or can be derived from a point cloud, etc. Thus, one goal of the pre-training phase is to obtain high-quality camera poses and a coarse-quality mesh with imperfect albedo textures. In examples, an albedo texture represents an image texture without shadows or highlights. Figure 6

[0102] ​As referred to herein, “mip” or “MIP” corresponds to multum in parvo, which means “much in little space.” In implementations, MIP mapping (e.g., “mipmapping”) is a technique in which an original high-resolution texture map is scaled and filtered into multiple resolutions within a texture file. MIP maps can be automatically generated based on the original texture, but can also be individually provided (e.g., painted or drawn) or adjusted by a user. In examples, each subsequent MIP level is half the size of the previous MIP level, which ensures that the complete texture (original texture and its MIP maps) is no more than 1.5 times (e.g., 150%) the original texture. Each scaled texture or MIP level can represent the appearance of the texture at a particular distance from the viewpoint of the user. As discussed further below, a given MIP map corresponds to a particular level of a specular environment map.

[0103] As referred to herein, a BRDF (bidirectional reflectance distribution function) is used to represent the radiative properties (e.g., reflective properties) of a surface and is a function of four real variables that define how light reflects on an opaque surface. Thus, a BRDF texture or map is a representation of such radiative or reflective properties of a surface.

[0104] As referred to herein, a roughness texture can be a grayscale image that represents the scattering of light on the surface of a 3D model, in which lower values of roughness (e.g., 0) correspond to shiny surfaces (e.g., glossy materials) that indicate less rough surfaces, and higher values (e.g., 1) correspond to a dull appearance (e.g., matte materials) that indicate rougher surfaces. In examples, a roughness texture can include values ranging from 0 to 1.

[0105] As referred to herein, an irradiance map texture includes a collection of samples of the total amount of light arriving at a particular location from each direction that the renderer can use to create diffuse lighting effects.

[0106] At the training phase, the computer client device 118 initializes the texture set 530:

[0107] 1. 3-channel BRDF texture - stores a texture of implicit BRDFs (1), shown as “BRDF”

[0108] 2. 1-channel roughness texture - stores a texture of implicit roughness parameters

[0109] 3. 3-channel irradiance map texture - stores a texture of implicit irradiance maps (2), shown as “Irradiance Map”

[0110] 4.3 3-channel specular MIP texture set - stores textures (3) of MIP levels of specular environment maps, shown as "Specular MIPs". In implementations, the format of the 3-channel specular MIP textures is the same as the format of the 3-channel radiance map textures. Storing these textures can be based on:

[0111] 1. All textures can be trainable and independent of each other

[0112] 2. Each texture is a blurred version of the previous texture

[0113] 3. In addition to (b), a trainable delta can be included in the corresponding MIP level

[0114] In examples, for the roughness texture, a value of 0 (e.g., roughness value) would result in sampling the environment map of the highest level (e.g., the highest MIP level from the 3-channel specular MIP texture set), while a value of 1 would result in sampling the environment map of the lowest level (e.g., the lowest MIP level from the 3-channel specular MIP texture set). For values between 0 and 1, a linear interpolation (e.g., trilinear) between the highest level and the lowest level is performed (e.g., any value below 0.5 is based on a linear interpolation of the highest level, and any value 0.5 and above is based on a linear interpolation of the lowest level, etc.).

[0115] In some implementations, training of the neural network model can be performed by the computer client device 118 or by a server (e.g., the interaction server system 110). During the training process, the following operations are performed:

[0116] 1. Initialize the textures described above to zero

[0117] 2. Sample a batch of images from the training data 510 (e.g., training set)

[0118] 3. Using the differentiable rasterizer 524, render the object as a rasterized RGB image 526 from the viewpoint 514 using the previously extracted mesh and textures. In examples, the differentiable rasterizer 524 performs shading for each pixel by stitching together the following values (e.g., generating a 9-channel texture such that each pixel has 9 values, where the 9-channel texture corresponds to the rasterized RGB image 526):

[0119] • BRDF value (treat texture (1) as albedo for the mesh)

[0120] • Radiance map value (query texture (3) by the surface normal (e.g., vector n in Figure 6

[0121] • Specular reflection value, by the surface reflection vector (e.g.,​Figure 6 The specular map is queried using the vector r) in the roughness texture. The corresponding specular MIP level is tri-linearly interpolated using the roughness value from the roughness texture. For example, if the roughness value is 0.3 and five MIP levels are provided (e.g., from the set of specular MIP textures), the second MIP level is blended together with the third MIP level.

[0122] 4. The rasterized RGB image 526 (e.g., 9-channel texture) from step (3) is fed to the input of a neural network model 528. In implementations, the neural network model 528 is an image-to-image model that receives an input image (e.g., the rasterized RGB image 526) and generates an output image corresponding to the predicted RGB image 520. In examples, the output image includes the same number of channels (e.g., 9) as the input image. However, it should be understood that the output image can be any number of channels (e.g., greater or fewer than the number of channels of the input image (e.g., 9)). Moreover, the set of view directions 516 (e.g., derived from the viewpoints 514 from the training data) are provided as inputs to the neural network model 528. In examples, the batch of images (e.g., from the training data 510) discussed above are used as ground truth to determine the loss (e.g., L2 loss 512) from the training process. Figure 6

[0123] 5. Next, the L2 loss 512 is determined, and the gradient 518 of the L2 loss 512 is also determined. The L2 loss 512 is determined using a loss function that minimizes the error of the sum of squared differences between the true values (e.g., the known correct outputs) and the predicted values. The gradient 518 can be understood as a vector of partial derivatives of a particular input value with respect to the objective function. In examples, the gradient is computed for each weight in the network model with backpropagation, and then the model weights of the neural network model 528 are updated using the gradient. Using backpropagation techniques, the neural network model 528 and all of the initialized textures ( (1), (2), (3), (4) from the set of textures 530) are optimized based on the gradient 518.

[0124]

[0125] After the training of the neural network model 528 is complete, a trained neural network model and a learned set of maps are generated (e.g., based on the above-optimized textures discussed in step 5 above). The trained neural network model and the learned set of maps can be provided to the computer client device 118 to perform an inference process.

[0126] ​​During the inference process, the computer client device 118 rasterizes an RGB image with the object mesh (e.g., the object mesh 522), a new viewpoint (e.g., a different vector than the vector associated with the viewpoint 514), and the learned set of maps, and passes the rasterized RGB image and the new viewpoint to the trained neural network for rendering of the object (e.g., corresponding to the object mesh), which then generates an output image corresponding to a neural rendered image of the object (e.g., having a reflective surface).

[0127] Figure 6 Example components used by the process flow described in FIG. 6 are shown in accordance with implementations of the subject technology. Figure 5

[0128] As shown, an object mesh (e.g., a 3D mesh) with texture sample points (e.g., obtained using a photogrammetry application) is provided.

[0129] In the example of FIG. 6, the computer client device 118 can sample the BRDF texture 630, the irradiance map texture 620, and the set of specular MIP textures 610 at the texture sample points of the object mesh. The vector w represents the view direction, the vector n represents the surface normal of the object, and the vector r represents the surface reflection vector corresponding to the set of specular MIP textures 610. Figure 6

[0130] In implementations, the computer client device 118 generates the irradiance map texture 620 based on the surface normal. In examples, the computer client device 118 estimates the surface normal based on the geometry of the surface (e.g., provided by the object mesh). Using interpolation, the computer client device 118 can determine another surface normal for any other point on the surface of the object mesh. The irradiance map texture 620 can be understood as an environment map (e.g., surrounding the object), and for any given vector, a sample of the irradiance map texture 620 can be obtained.

[0131] In examples, the computer client device 118 samples the BRDF texture 630 at the texture sample points, which obtains an RGB value. In examples, the computer client device 118 samples the irradiance map texture 620 at the texture sample points, which obtains another RGB value. In examples, using the roughness texture value at the texture sample point (discussed above in Figure 5 In examples, the computer client device 118 samples a particular level from the set of specular MIP textures 610 (as discussed previously in Figure 5 In implementations, the three RGB values are stacked (e.g., stitched) together as discussed previously.

[0132] Figure 7 ​​Examples of irradiance maps and specular maps are shown in accordance with implementations of the subject technology.

[0133] As shown, the irradiance map texture 710 is an RGB texture learned from training of the neural network model 528 as discussed above. If sampled from the irradiance map texture 710 using surface normals (e.g., from the surface normals 712), the integrated environment is determined. As further shown, the loss during training is determined with the image 720 corresponding to the ground truth, and the image 730 corresponds to the learned specular MIP texture at the highest level (e.g., at the highest level, the roughness texture has a value of 0 at the texture sample point). Figure 6

[0134] Figure 8 Examples of neural rendering and neural rendering with glossy (e.g., reflective) surface techniques are shown in accordance with some implementations of the subject technology.

[0135] As shown, the rendered image 810 represents an image using neural rendering that does not support rendering of reflective (e.g., glossy) surfaces, and the rendered image 820 represents an image using neural rendering that supports reflective surfaces (e.g., using the trained neural network model discussed earlier). As can be observed, the rendered image 820 includes a more realistic rendering of the reflective surfaces (e.g., a greater amount of detail and clarity) when compared to the rendered image 810.

[0136] Figure 9 is a flow diagram showing a method in accordance with certain example implementations. The method can be embodied in computer-readable instructions for execution by one or more computer processors, such that the operations of the method can be performed, in part or in whole, by the computer client device 118. However, it should be understood that at least some of the operations of the method can be deployed on various other hardware configurations, and the method is not intended to be limited to the computer client device 118 or any of the components or systems mentioned above.

[0137] ​According to some examples, at operation 902, the computer client device 118 receives an object mesh, information related to a viewpoint for rendering an image of an object having a reflective surface, and a set of maps. In implementations, the object mesh is generated based on photogrammetry techniques applied on a set of source images of a physical object, the set of source images including different images captured from a set of different viewpoints of the physical object. Further, the viewpoint includes a vector from a point of a surface of the object mesh, and the set of maps includes a set of textures modified according to a trained neural network model, the set of textures including a BRDF (bidirectional reflectance distribution function) texture, a roughness texture, an irradiance map texture, and a set of specular textures. In implementations, the roughness texture includes a set of pixels each having a value from 0 to 1. In implementations, the set of specular textures includes a plurality of specular textures including at least a first specular texture and a second specular texture, the second specular texture being half the size of the first specular texture. Further, the BRDF texture includes a first three-channel image, the roughness texture includes a one-channel image, the irradiance map texture includes a second three-channel image, and each specular texture includes a particular three-channel image corresponding to a particular level of environment map.

[0138] At operation 904, the computer client device 118 generates a rasterized RGB (red green blue) image based on the object mesh, the viewpoint, and the set of maps. In implementations, generating the rasterized RGB image using a differentiable rasterizer includes generating each pixel of the rasterized RGB image by stitching a first set of values of a first pixel of a first three-channel image of the BRDF texture, a second set of values of a second pixel of a second three-channel image of the irradiance map texture, and a third set of values of a third pixel from a particular specular texture of the set of specular textures. In examples, the second set of values is determined by querying the irradiance map texture using a surface normal vector, and the third set of values is determined by querying the particular specular texture by a surface reflection vector.

[0139] In implementations, the computer client device 118 sends the rasterized RGB image to a neural network model for rendering the object. In implementations, the neural network model includes an image-to-image model.

[0140] At operation 906, the computer client device 118 generates, using the neural network model, an output image of the object having the reflective surface based at least in part on the rasterized RGB image and the viewpoint.

[0141] At operation 908, the computer client device 118 provides for displaying the output image of the object having the reflective surface on a display of the computer client device.

[0142] Machine Architecture

[0143] Figure 10is a diagrammatic representation of a machine 1000 within which instructions 1002 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine 1000 to perform any one or more of the methodologies discussed herein can be executed. For example, the instructions 1002 can cause the machine 1000 to execute any one or more of the methods described herein. The instructions 1002 transform the general, non-programmed machine 1000 into a particular machine 1000 programmed to carry out the described and illustrated functions in the manner described. The machine 1000 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1000 can operate in the capacity of a server machine or a client machine in server-client network environments, or as a peer machine in peer-to-peer (or distributed) network environments. The machine 1000 can comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions 1002, sequentially or otherwise, that specify actions to be taken by machine 1000. Further, while only a single machine 1000 is illustrated, the term “machine” shall also be taken to include a collection of machines that individually or jointly execute the instructions 1002 to perform any one or more of the methodologies discussed herein. For example, the machine 1000 can include the user system 102 or any of the multiple server devices that form part of the interaction server system 110. In some examples, the machine 1000 can further include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed in the server side, and certain operations of the particular method or algorithm are performed in the client side.

[0144] The machine 1000 can include processors 1004, memory 1006, and input / output (I / O) components 1008, which can be configured to communicate with each other via a bus 1010. In an example, the processors 1004 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), another processor, or combinations of any of these processors) can include, for example, a processor 1012 and a processor 1014 that execute instructions 1002. The term “processor” is intended to include multiple processors that can be configured to execute instructions 1002. Although Figure 10 Multiple processors 1004 are illustrated, but the machine 1000 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or combinations of any of these.

[0145] The memory 1006 includes a main memory 1016, a static memory 1018, and a storage unit 1020, each of which can be accessed via the bus 1010 by the processors 1004. The main memory 1016, static memory 1018, and storage unit 1020 store the instructions 1002 that implement any one or more of the methods or functions described herein. The instructions 1002 can also reside completely, or a portion thereof, within the main memory 1016, within the static memory 1018, within the storage unit 1020, within at least one of the processors 1004 (e.g., within the cache memory of the processors), or any suitable combination thereof, during execution thereof by the machine 1000.

[0146] The I / O components 1008 can include various components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so forth. The specific I / O components 1008 that are included in the machine 1000 will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I / O components 1008 can include Figure 10Many other components not shown in FIG. 10 can also be included. In various examples, the I / O components 1008 can include user output components 1024 and user input components 1026. The user output components 1024 can include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The user input components 1026 can include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., a physical button, a touch screen that provides location and force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0147] In further examples, the I / O components 1008 can include biometric components 1028, motion components 1030, environmental components 1032, or positioning components 1034, among a myriad of other components. For example, the biometric components 1028 include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like.

[0148] The environmental components 1032 include, for example, one or more cameras (with still and / or video capabilities), illumination sensors (e.g., photometers), temperature sensors (e.g., one or more thermometers that detect ambient temperature), humidity sensors, pressure sensors (e.g., barometers), acoustic sensors (e.g., one or more microphones that detect background noise), proximity sensors (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors to detect concentrations of hazardous gases for safety or to measure pollutants in ambient air), or other components that can provide indications, measurements, or signals corresponding to a physical environment.

[0149] With respect to cameras, the user system 102 can have a camera system that includes, for example, a front-facing camera on a front surface of the user system 102 and a rear-facing camera on a rear surface of the user system 102. The front-facing camera can be used, for example, to capture still images and videos of a user of the user system 102 (e.g., “selfies”), which can then be augmented with the above-described augmentation data (e.g., filters). The rear-facing camera can be used, for example, to capture still images and videos in a more conventional camera mode, which are similarly augmented with augmentation data. In addition to front- and rear-facing cameras, the user system 102 can include a 360° camera for capturing 360° photos and videos.

[0150] In addition, the camera system of the user system 102 can include dual rear-facing cameras (e.g., a primary camera and a depth-sensing camera), or even a triple, quadruple, or quintuple rear-facing camera configuration on the front and rear sides of the user system 102. These multi-camera systems can include, for example, a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0151] The positioning components 1034 include location sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and the like.

[0152] Communication can be implemented using a wide variety of technologies. The I / O The communication components 1036 can include wired communication Bluetooth® low energy, components, and other communication components to provide communication via other modalities. The devices 1040 can be another machine or any of a wide array of peripheral devices (e.g., a peripheral device coupled via USB).

[0153] Moreover, the communication components 1036 can detect identifiers or include components operable to detect identifiers. For example, the communication components 1036 can include radio frequency identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., an optical sensor to detect one-dimensional bar codes such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes), or acoustic detection components (e.g., microphones to identify tagged audio signals). In addition, a variety of information can be derived via the communication components 1036, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via cellular signal triangulation, location via detecting NFC beacon signals that can indicate a particular location, and so forth.

[0154] The various memories (e.g., main memory 1016, static memory 1018, and memory of processor 1004) and storage unit 1020 can store one or more sets of instructions and data structures (e.g., software) embodying or used by any one or more of the methodologies or functions described herein. These instructions (e.g., instructions 1002), when executed by processor 1004, cause various operations to implement the disclosed examples.

[0155] The instructions 1002 can be transmitted or received by network interface device (e.g., network interface component included in communication components 1036) using a transmission medium via the network 1038 and using any one of a number of well-known transfer protocols (e.g., hypertext transfer protocol (HTTP)). Similarly, instructions 1002 can be transmitted or received using a transmission medium via the coupling (e.g., peer-to-peer coupling) to device 1040.

[0156] Software Architecture

[0157] Figure 11is a block diagram 1100 illustrating software architecture 1102, which can be installed on any one or more of the devices described herein. The software architecture 1102 is

[0158] The operating system 1112 manages hardware resources and provides common services. The operating system 1112 includes, for example, a kernel 1124, services 1126, and drivers 1128. The kernel 1124 acts as an abstraction layer between the hardware and the other software layers. For example, the kernel 1124 provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functionality. The services 1126 can provide other common or The drivers 1128 are responsible for controlling or interfacing with the underlying hardware. For instance, the drivers 1128 can include display drivers, camera drivers, Bluetooth® or Bluetooth® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), WI-FI® or WI-FI® Direct drivers, audio drivers, power management drivers, and so forth.

[0159] The libraries 1114 provide a higher-level common infrastructure that can be used by the applications 1118. For example, libraries 1114 can include system libraries 1130 (e.g., C standard library, which provides function declarations for ​

[0160] The framework 1116 provides common high-level infrastructure used by the applications 1118. For example, the framework 1116 provides various graphical user interface (GUI) functions, high-level resource management, and high-level location services. The framework 1116 can provide a broad spectrum of other APIs that can be used by the applications 1118, some of which are specific to a particular operating system or platform.

[0161] In an example, the applications 1118 include a home application 1136, a contacts application 1138, a browser application 1140, a book reader application 1142, a location application 1144, a media application 1146, a messaging application 1148, a game application 1150, and a broad assortment of other applications such as a third-party application 1152. The applications 1118 are programs that execute functions defined in the programs. Programs can be created using a variety of programming languages, such as object-oriented TM programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C or assembly language). In a specific example, the third-party application 1152 (e.g., an application developed by an entity other than the vendor of the particular platform) can be a mobile software running on a mobile operating system such as the IOS TM operating system or other mobile operating systems. In this example, the third-party application 1152 can invoke the API calls 1120 provided by the operating system 1112 to, for example, have the mobile device perform functions described herein. TM TM

[0162] Glossary

[0163] “Carrier signal” means, for example, any intangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine and includes digital or analog communications signals. The instructions can be transmitted or received over a network via the network interface device using a transmission medium.

[0164] “Client device” means, for example, any machine that interfaces to a communications network to obtain resources from one or more server systems or other client devices. A client device can be, but is not limited to, a mobile phone, desktop computer, laptop computer, portable digital assistants (PDAs), smart phones, tablets, ultrabooks, netbooks, laptops, multi-processor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access a network. ​​​

[0165] “communication network” refers to one or more portions of a network, which can be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a network, another type of network, or a combination of two or more such networks. For example, a network or a portion of a network can include a wireless or cellular network, and the coupling can be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, the coupling can implement any of a variety of types of data transfer techniques for wired or wireless communication, such as Single Carrier Radio Transmission Technology (lxRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, third Generation Partnership Project (3GPP) including 3G, fourth generation wireless (4G) networks, Universal Mobile

[0166] A "component" refers to, for example, a logical, physical, or logical and physical grouping of resources, physical or logical entities, or a combination of resources and physical or logical entities, with boundaries that are defined by a functional or subroutine call, a leap of control, an API, or other technological means for providing partitions or modularity of a particular processing or control function. A component can be combined with other components to perform a machine process, via the interfaces with other components. A component can be a packaged functional hardware unit designed for use with other components, and to perform a particular function relating to a larger function, sometimes referred to as a subfunction. A component can be a part of a program that performs a specific function and is typically invoked by other components of the program. Components can constitute either software components (e.g., code embodied on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit capable of performing certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or application portion) as a hardware component that operates to perform certain operations as described herein. A hardware component can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component can be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component can also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, hardware components become specific machines (or specific components of a machine) uniquely tailored to perform the configured functions and are no longer general-purpose processors. It will be appreciated that theWhere multiple hardware components are present, communication can be achieved through signal transmission among or between two or more of the hardware components (e.g., through appropriate circuits and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can for example be achieved by storing information in memory structures to which the multiple hardware components have access, and by retrieving information from the memory structures. For example, one hardware component can perform an operation and store output of that operation in a memory device to which it is communicatively coupled. A further hardware component can then, at a later time, access the memory device to retrieve and process the stored output. Hardware components can also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information). The various operations of example methods described herein can be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, “processor- implemented component” refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be at least partially processor-implemented, with a particular processor or processors being an example of hardware. For example, at least some of the operations of a method can be performed by one or more processors or processor-implemented components. Moreover, a processor or processors can also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations can be performed by a group of computers (as examples of machines including processors), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., an API). The performance of certain of the operations can be distributed among the processors, not only residing within a single machine, but deployed across a number of machines. In some examples, the processors or processor-implemented components can be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other examples, the processors or processor-implemented components can be distributed across a number of geographic locations.

[0167] “Computer-readable storage medium” refers to both machine-storage media and transmission media. Thus, the terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” mean the same thing and can be used interchangeably in this disclosure.

[0168] A "ephemeral message" refers to a message that is accessible, for example, for a time-limited duration. An ephemeral message can be text, image, video, etc. The access time for an ephemeral message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is temporary.

[0169] "Machine-storage medium" refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be taken to include, but not be limited to, solid-state memories, as well as optical and magnetic media, including memory internal or external to a processor. Specific examples of machine-storage media, computer-storage media, and device-storage media include non-volatile memory, including by way of example semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine-storage medium," "device-storage medium," "computer-storage medium," and "device-storage medium" mean the same thing and can be used interchangeably in this disclosure. The terms "machine-storage medium," "computer-storage medium," and "device-storage medium" explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered under the term "signal medium."

[0170] "Non-transitory computer-readable storage medium" refers to, for example, a tangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine.

[0171] "Signal medium" refers to any intangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of software or data. The term "signal medium" shall be taken to include any form of a modulated data signal, carrier wave, and so on. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and can be used interchangeably in this disclosure.

[0172] "User device" refers to, for example, a device that is accessed by, controlled by, or owned by a user and with which the user interacts to perform actions or interactions with other users or computer systems.

Claims

1. A method comprising: receiving an object mesh, information related to a viewpoint for rendering an image of an object having a reflective surface, and a set of maps; generating a rasterized RGB (red green blue) image based on the object mesh, the viewpoint, and the set of maps; generating, using a neural network model, an output image of the object having the reflective surface based at least in part on the rasterized RGB image and the viewpoint; and providing for display of the output image of the object having the reflective surface on a display of a computer client device. generating the object mesh based on photogrammetry techniques applied on a set of source images of a physical object, the set of source images including images captured from a set of different viewpoints of the physical object, and further including:

2. The method of claim 1, wherein, sending the rasterized RGB image to the neural network model for rendering the object. the viewpoint includes a vector from a point of a surface of the object mesh, and the set of maps includes a set of textures modified according to training the neural network model, the set of textures including a BRDF (bidirectional reflectance distribution function) texture, a roughness texture, an irradiance map texture, and a set of specular textures.

3. The method of claim 1, wherein, the roughness texture includes a set of pixels each having a value from 0 to 1.

4. The method of claim 3, wherein, the set of specular textures includes a plurality of specular textures including at least a first specular texture and a second specular texture, the second specular texture being half the size of the first specular texture.

5. The method of claim 3, wherein, the BRDF texture includes a first three-channel image, the roughness texture includes a one-channel image, the irradiance map texture includes a second three-channel image, and each specular texture includes a particular three-channel image corresponding to a particular level of environment map.

6. The method of claim 3, wherein, generating the rasterized RGB image includes:

7. The method of claim 6, wherein, generating each pixel of the rasterized RGB image by concatenating a first set of values of a first pixel of the first three-channel image of the BRDF texture, a second set of values of a second pixel of the second three-channel image of the irradiance map texture, and a third set of values of a third pixel from a particular specular texture of the set of specular textures. determining the second set of values by querying the irradiance map texture using a surface normal vector.

8. The method of claim 7, wherein, determining the third set of values by querying the particular specular texture by means of a surface reflection vector.

9. The method of claim 7, wherein, the neural network model includes an image-to-image model.

10. The method of claim 1, wherein, 11. A system comprising: a processor; and a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising: receiving an object mesh, information related to a viewpoint for rendering an image of an object having a reflective surface, and a set of maps; generating a rasterized RGB (red green blue) image based on the object mesh, the viewpoint, and the set of maps; generating, using a neural network model, an output image of the object having the reflective surface based at least in part on the rasterized RGB image and the viewpoint; and providing for display of the output image of the object having the reflective surface on a display of a computer client device. ​ ​ 12. The system of claim 11, wherein, generating the object mesh based on photogrammetry techniques applied on a set of source images of a physical object, the set of source images including images captured from a set of different viewpoints of the physical object, and further including: sending the rasterized RGB image to the neural network model for rendering the object.

13. The system of claim 11, wherein, the viewpoints include vectors from points of a surface of the object mesh, and the set of maps includes a set of textures modified according to training the neural network model, the set of textures including a BRDF (bidirectional reflectance distribution function) texture, a roughness texture, an irradiance map texture, and a specular texture set.

14. The system of claim 13, wherein, the roughness texture includes a set of pixels, each pixel having a value from 0 to 1.

15. The system of claim 13, wherein, the specular texture set includes a plurality of specular textures, the plurality of specular textures including at least a first specular texture and a second specular texture, the second specular texture being half the size of the first specular texture.

16. The system of claim 13, wherein, the BRDF texture includes a first three-channel image, the roughness texture includes a one-channel image, the irradiance map texture includes a second three-channel image, and each specular texture includes a particular three-channel image corresponding to a particular level of environment map.

17. The system of claim 16, wherein, generating the rasterized RGB image includes: generating each pixel of the rasterized RGB image by concatenating a first set of values of a first pixel of the first three-channel image of the BRDF texture, a second set of values of a second pixel of the second three-channel image of the irradiance map texture, and a third set of values of a third pixel from a particular specular texture of the set of specular textures.

18. The system of claim 17, wherein, determining the second set of values by querying the irradiance map texture using a surface normal vector.

19. The system of claim 17, wherein, determining the third set of values by querying the particular specular texture by means of a surface reflection vector.

20. A non-transitory computer-readable medium including instructions that, when executed by a computing device, cause the computing device to perform operations comprising: receiving an object mesh, information related to a viewpoint for rendering an image of an object having a reflective surface, and a set of maps; generating a rasterized RGB (red-green-blue) image based on the object mesh, the viewpoint, and the set of maps; generating, using a neural network model, an output image of the object having the reflective surface based at least in part on the rasterized RGB image and the viewpoint; and providing for display of the output image of the object having the reflective surface on a display of a computer client device.

Images