Generating Augmented Reality Pre-Renderings Using Template Images

Abstract

To provide systems and methods for generating augmented reality prerenderings.SOLUTION: Systems and methods for generating augmented reality prerenderings according to an embodiment can provide a benefit of an augmented reality rendering without requiring the use of user data. Template images can be used instead of user data to protect the user's privacy while enabling the user to see an object or product rendered onto a preferred template image or a variety of template images.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] Related Applications This application claims priority to and the benefit of U.S. Non-Provisional Patent Application No. 17 / 153,263, filed January 20, 2021. U.S. Non-Provisional Patent Application No. 17 / 153,263 is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to pre-rendering images and, more particularly, to pre-rendering augmented reality images from a set of template images to enable a limited augmented reality experience from the template images, for example, as an alternative to a real-time personalized augmented reality experience. [Background technology]

[0003] Augmented reality (AR) can refer to the creation and execution of interactive experiences of real-world environments in which objects present in the real world are augmented with computer-generated perceptual information. As an example, an AR experience can include augmenting a scene captured by a camera by inserting virtual objects into the scene and / or modifying the appearance of real-world objects included in the scene.

[0004] When searching for aesthetically-intense products, such as cosmetics, oftentimes it is not enough to simply look at the product packaging or even the product itself. To solve this problem, efforts have been made to digitize cosmetics and other products in augmented reality (AR), allowing consumers to visualize the product on themselves or within their personal environment. However, many users may not want to use AR try-on due to natural friction. For example, the user may not be in a location where using a camera is practical, the user may not feel they look their best, may be reluctant to turn on the camera, and / or the user may simply not want to grant camera permission.

[0005] Additionally, live AR experiences can require significant data bandwidth and processing power. Summary of the Invention [Means for solving the problem]

[0006] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0007] One exemplary aspect of the present disclosure is directed to a computer-implemented method for providing pre-rendered augmented images. The method can include obtaining, by a computing device, a plurality of template images. The method can include processing, by the computing device, the plurality of template images with an augmented reality rendering model to generate a plurality of pre-rendered images. In some implementations, the method can include receiving, by the computing device, a request and preferences for a result image. The method can include providing, by the computing device, a pre-rendered result based at least in part on the request and preferences. In some implementations, the pre-rendered result can be a pre-rendered image from the plurality of pre-rendered images.

[0008] Another example aspect of the present disclosure is directed to a computing system. The computing system may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include obtaining an augmented reality asset. In some implementations, the augmented reality asset may include digitization parameters. The operations may include obtaining a plurality of template images. The operations may include processing the plurality of template images with the augmented reality model to generate a plurality of pre-rendered images based at least in part on the digitization parameters, and storing the plurality of pre-rendered images.

[0009] Another example aspect of the present disclosure is directed to one or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more processors, cause a computing system to perform operations. The operations can include obtaining an augmented reality asset. The augmented reality asset can include digitization parameters. In some implementations, the operations can include obtaining a plurality of template images. The operations can include processing the plurality of template images with the augmented reality model to generate a plurality of pre-rendered images based at least in part on the digitization parameters. The operations can include storing the plurality of pre-rendered images on a server and receiving a search query that can include one or more search terms. The one or more search terms can be related to a product. In some implementations, the operations can include providing search results. The search results can include pre-rendered images from the plurality of pre-rendered images retrieved from the server. The pre-rendered images can include renderings of the product.

[0010] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0011] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain the associated principles.

[0012] A detailed description of embodiments, directed to those skilled in the art, is set forth herein with reference to the accompanying drawings. [Brief explanation of the drawings]

[0013] [Figure 1A] FIG. 1 is a block diagram of an exemplary computing system that performs pre-rendering, according to an exemplary embodiment of the present disclosure. [Figure 1B] FIG. 1 is a block diagram of an exemplary computing device that performs pre-rendering, according to an exemplary embodiment of the present disclosure. [Figure 1C] FIG. 1 is a block diagram of an exemplary computing device that performs pre-rendering, according to an exemplary embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram of an exemplary display of a pre-rendered image, according to an exemplary embodiment of the present disclosure. [Figure 3] FIG. 2 is a block diagram of an exemplary display of a pre-rendered image, according to an exemplary embodiment of the present disclosure. [Figure 4] FIG. 2 is a block diagram of an exemplary display of a pre-rendered image, according to an exemplary embodiment of the present disclosure. [Figure 5] FIG. 1 is a block diagram of an exemplary pre-rendering system, according to an exemplary embodiment of the present disclosure. [Figure 6] FIG. 1 is a flowchart diagram of an example method for performing pre-rendering, according to an example embodiment of the present disclosure. [Figure 7] FIG. 1 is a flowchart diagram of an example method for performing pre-rendering, according to an example embodiment of the present disclosure. [Figure 8] FIG. 1 is a flowchart diagram of an example method for performing pre-rendering, according to an example embodiment of the present disclosure. [Figure 9] FIG. 1 is a block diagram of an exemplary pre-rendering system, according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] Reference numbers that are repeated among the drawings are intended to identify like features in various implementations.

[0015] Overview In general, the present disclosure is directed to systems and methods for pre-rendering augmented reality template images to enable limited augmented reality experiences from the template images, for example, as an alternative to real-time personalized augmented reality experiences using a user's specific images. In some implementations, the systems and methods can be implemented as a platform for generating pre-rendered augmented reality images from a set of template images. The system can include obtaining augmented reality assets and a set of template images. The template images can be a diverse set of images depicting people or settings with diverse characteristics or features, such as different eye colors or other visual characteristics. The set of template images can be processed by an augmented or augmented reality model parameterized by the obtained augmented reality assets. The augmented reality model can output a set of pre-rendered images, which can be stored on a server. In some implementations, the system can receive a request for result images along with user preferences. For example, the request can be a search query in the form of one or more search terms. The system can process the request and preferences to determine result images. The result images can be one or more pre-rendered images from the set of pre-rendered images. For example, a user can select (or preselect) one or more preferences, which can guide the selection of one or more pre-rendered images as result images. The results can be provided to the user. In this manner, the user can be provided with one or more pre-rendered images that satisfy the user's preferences (e.g., pre-rendered images generated from templates having characteristics that match the user's characteristics). Thus, in some cases, a user can see how a product would appear on a person or setting similar to themselves, but without having to be in a location where using a camera would be practical and / or without having to supply actual images of themselves or their environment.In some implementations, the system may also provide the user with a link to the real-time augmented reality experience, for example, so that the user can continue to pursue the full real-time AR experience if desired.

[0016] In some implementations, pre-rendering of augmented reality images can be facilitated by a platform. The platform can be used to collect augmented reality data assets created using the platform, or alternatively, can collect augmented reality assets generated outside the platform. The platform can collect augmented reality assets from third-party companies offering products for sale (e.g., cosmetics (e.g., lipstick, eye shadow, etc.), furniture or other household items (e.g., electronics, cookware, glassware, decorative items, plants, etc.), clothing, paint colors, automobiles, various electronic devices, etc.). Additionally, the platform can use the collected augmented reality assets to render rendering effects into template images to generate pre-rendered images. The template images can be stored locally or obtained from outside the platform. The pre-rendered images can then be stored for later display to the user. For example, a user computing device can send a request for one or more pre-rendered images. The request can be a search query, a user selection, or an automated request in response to a user action. The platform can process the request along with preferences to provide one or more pre-rendered images related to the request and the user preferences.

[0017] The systems and methods can include obtaining a plurality of template images. The plurality of template images can be a set of images depicting a similar focus but with differences (e.g., the template images each depict a living room, but each photo depicts the room in a different color scheme, with furniture and decor having different themes and colors). The systems and methods can process the plurality of template images with an augmented reality rendering model to generate a plurality of pre-rendered images. In some implementations, the augmented reality rendering model can include object tracking and rendering. The plurality of pre-rendered images can be a set of augmented images, where each template image can be augmented to include augmented reality rendering effects. The augmented reality rendering effects can be products sold by a third party, where a consumer can virtually “try on” the product in different template scenarios (e.g., furniture items rendered within the template room image). In some implementations, the systems and methods can include receiving a request and preference for result images. The request can come from a user computing system, where the user indicates a desire to view pre-rendered images. In some implementations, the request can include search terms entered into a search engine. The preferences can include selections made at the time of the request or previously stored that can indicate a preferred template type or a preferred template. The systems and methods can then provide a pre-rendered result based at least in part on the request and the preferences, where the pre-rendered result can be a pre-rendered image from a plurality of pre-rendered images. In some implementations, the pre-rendered result can be a pre-rendered image that matches the preferences, where the preferences include a template selected by the user from a plurality of template images.In some implementations, the systems and methods may provide a link to an augmented reality rendering experience for live try-on.

[0018] In some implementations, systems and methods can obtain augmented reality assets, which can be stored for use by the augmented reality model or augmented model. The augmented reality assets can include digitization parameters. The digitization parameters can enable the augmented reality model or augmented model to render a particular rendering effect. In some implementations, the augmented reality assets can be used by the augmented reality model to process a set of template images to generate a set of pre-rendered images. The set of pre-rendered images can then be stored for later retrieval. In some implementations, the set of pre-rendered images can be stored on a server.

[0019] In some implementations, the set of template images can be processed by object-tracking computer vision algorithms and computer rendering operations to generate pre-rendered augmented reality images that mimic the appearance of the product. The product can be applied to a depicted face in the template images or inserted into a depicted image of the set of template images. The rendering operations can be influenced by CPU or GPU algorithms and corresponding parameters. The corresponding parameters can be parameters that realistically capture the appearance of the product inserted into the template images under the depicted lighting conditions.

[0020] In some implementations, the systems and methods can include receiving a search query including one or more search terms, where the one or more search terms are related to a product. Further, the systems and methods can include providing search results. The search results can include a pre-rendered image from a plurality of pre-rendered images. Further, the pre-rendered image can include a rendering of the product.

[0021] In some implementations, the set of template images can be processed to generate a set of template models that can be processed by the augmented reality model or augmented model. In some implementations, the template models can be modified before being processed. The systems and methods can receive input to modify a template model of the plurality of template models. The systems and methods can modify the template model based at least in part on the input. In some implementations, the systems and methods can include providing a template model of the plurality of template models for display.

[0022] In some implementations, an augmented reality model may include a perceptual subgraph and a rendering subgraph. The perceptual subgraph may be uniform across the system. The perceptual subgraph may be used with various different rendering subgraphs. The rendering subgraph may be constructed by a third party to generate the rendering effect provided to the user. The rendering subgraph may be constructed and then used by an augmented reality pre-rendering platform that stores the perceptual subgraph. The rendering subgraph may differ depending on the rendering effect and the third party. In some implementations, a single perceptual subgraph may be used with multiple rendering subgraphs to render multiple renderings in an augmented image or video. For example, a photo or video of a face may be processed to generate an augmented reality rendering of lipstick, eye shadow, and mascara on the face. The processing may include a single perceptual subgraph but may include rendering subgraphs for each product (i.e., lipstick, eye shadow, and mascara).

[0023] The systems and methods disclosed herein can be applicable to a variety of augmented reality experiences (e.g., household goods, makeup, automotive (3D), eyeglasses, jewelry, clothing, and haircuts). For example, the systems and methods disclosed herein can be used to generate a set of pre-rendered images to provide consumers with renderings of products in various environments or applications. In some implementations, the template images can be various room images with different setups, color schemes, and decorations. The desired rendering effect can be a sofa. The system can import the set of template images and augmented reality assets related to the sofa and generate a set of pre-rendered images to provide to the user. The pre-rendered images can include an augmented reality rendering of a desired sofa in each of the template images to allow the user to see how the sofa might look in various room types with different color schemes, decorations, and layouts.

[0024] In some implementations, the multiple template images can include a set of facial images with diverse characteristics or features, such as, for example, different eye colors or other visual characteristics, whereby a consumer can select and / or store a preferred template that may most resemble them. The rendering effect can be a makeup rendering (e.g., lipstick, eye shadow, mascara, foundation, etc.). The augmented reality model can process the augmented reality assets and the set of template images to generate pre-rendered makeup images, whereby each of the multiple template images can be augmented to include makeup products. For example, a particular shade of lipstick can be rendered within each of the template images.

[0025] In some implementations, a set of template images can be manually or automatically selected from a large corpus of images to provide a representative group of template images that show representative images of various scenarios or features for a given topic. For example, a topic can be a room, a neighbor, a face, etc.

[0026] The systems and methods disclosed herein can be used to provide personalized advertisements to users. For example, user preferences can be stored. The stored preferences allow the provided product advertisements to be adapted to render the advertised products on the user's preferred template images.

[0027] In some implementations, the augmented reality assets can be managed, generated, and / or reproduced by the product brand. Digitization parameters can be imported from third-party companies. In some implementations, the digitization parameters can be extracted from a third-party rendering engine or exported from a provided template for generating augmented reality rendering effects.

[0028] Additionally, the perceptual model can be adjusted manually or automatically to provide an optimized mesh. The adjustments can be responsive to lighting or varying image quality. The platform may provide a preview to assist in modifying the augmented reality model. In some implementations, the platform may include a pipeline for retrieving images.

[0029] The pre-rendered images may include tagging to index the rendered product.

[0030] In some implementations, the indexed products may include a consistent naming scheme for products and product colors to improve search results. The platform may include a data structure that can relate augmented reality assets to specific semantic or lexicographic entities. This data structure may be useful for understanding search queries to create product mappings.

[0031] In some implementations, the pre-rendered images may be provided as a carousel for the user to scroll through. Alternatively, various pre-rendered images with different product renderings but the same template image may be provided in the form of a carousel for personalized previews. Additionally, in some implementations, the carousel may include a virtual "try-on" panel to provide the user with an augmented reality rendering experience that processes the user's data to provide user-augmented images or video.

[0032] In some implementations, the augmented reality platform may retrieve data assets for rendering augmented reality effects via systems and methods for data asset acquisition. The systems and methods for data asset acquisition may involve one or more systems or devices. The first computing device may be a server, a facilitating computing device, or an intermediary computing device. The second computing device may be a third-party computing device. The third party may be a video game company, a product manufacturer, or a product brand. The first computing device and the second computing device may exchange data to generate an augmented reality rendering experience for the user. The augmented reality rendering experience may include rendering an augmented reality view including one or more products or items. The products may be cosmetics (e.g., lipstick, eye shadow, etc.), furniture or other household items (e.g., electronics, cookware, glassware, decorations, plants, etc.), clothing, paint colors, automobiles, various electronic devices, or other items.

[0033] Obtaining the data assets may include the first computing device sending a software development kit to the second computing device. The software development kit may include templates for building rendering effect shaders. The software development kit may include example effects, tools for building the rendering effect, and a preview mode useful for building augmented reality renderings. The second computing device may be used to build the rendering effect, and once the rendering effect is built, the second computing device may export the built rendering effect data to a renderable compressed file (e.g., a .ZIP file) that may include the data assets necessary to recreate the rendering effect. The data assets may then be sent to the first computing device. Upon receiving the data assets, the first computing device may store the data assets for use in an augmented reality rendering experience provided to a user. The provided augmented reality rendering experience may be provided to a user, who may input user data for processing, and the output may be augmented user data including the rendering effect built on the second computing device. The user data may be image data or video data captured by the user device. In some implementations, the user data may be a live camera feed.

[0034] Additionally, in some implementations, the system and method can be used as a visual compatibility calculator. For example, the system and method can be used to ensure that a particular product or part will fit a desired space or location. The system and method can be used to virtually test the dimensions / size of a product using virtual reality. A third party can provide a data asset that can include data describing the measurements of a product. The data asset can then be used to provide a user with an augmented reality rendering experience in which the product is rendered according to the measurements provided by the third party. This aspect allows consumers to "try on" a product to visualize the space it will occupy.

[0035] The systems and methods of the present disclosure provide many technical effects and advantages. As one example, the systems and methods can generate a set of pre-rendered images using a set of template images and an augmented reality experience. The systems and methods can also be used to provide consumers with pre-rendered images of products on various templates. Furthermore, the systems and methods can enable consumers to view pre-rendered augmented reality renderings on templates that meet the consumer's preferences.

[0036] Another technical advantage of the systems and methods of the present disclosure is their ability to provide "try-on" images that match consumer shopping preferences when an augmented reality real-time try-on experience is less than ideal.

[0037] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the drawings.

[0038] Exemplary Devices and Systems 1A illustrates a block diagram of an exemplary computing system 100 for performing pre-rendering, according to an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0039] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0040] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operably connected processors. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0041] In some implementations, the user computing device 102 can store or include one or more augmented reality models 120. For example, the augmented reality models 120 can be or otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models including nonlinear and / or linear models. The neural networks can include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Exemplary augmented reality rendering models 120 are described with reference to FIGS. 5-9.

[0042] In some implementations, one or more augmented reality rendering models 120 may be received from a server computing system 130 over a network 180, stored in a user computing device memory 114, and then used or implemented by one or more processors 112. In some implementations, a user computing device 102 may implement multiple parallel instances of a single augmented reality rendering model 120.

[0043] More specifically, the augmented reality rendering model can utilize a perceptual model and a rendering model to render the augmented reality rendering within the template image. The perceptual model can be a model stored on the platform that is applicable with various rendering models. The rendering model can be generated by a third party using a software development kit. The rendering model can be constructed by a third party and then sent to the platform. In some implementations, the platform can receive data assets for the rendering model from a third party.

[0044] The template image may be processed by an augmented reality rendering model along with a perceptual model that generates a mesh and a segmentation mask based on processing of the template image, and the rendering model may process the template image, mesh, and segmentation mask to generate a pre-rendered image.

[0045] Additionally or alternatively, one or more augmented reality rendering models 140 may be included in or stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the augmented reality rendering models 140 may be implemented by the server computing system 130 as part of a web service (e.g., a "pre-rendered try-on" service). Thus, one or more models 120 may be stored and implemented on the user computing device 102 and / or one or more models 140 may be stored and implemented on the server computing system 130.

[0046] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that senses the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component functions to implement a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0047] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operably connected processors. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0048] In some implementations, server computing system 130 includes or is implemented by one or more server computing devices. When server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a serial computing architecture, a parallel computing architecture, or some combination thereof.

[0049] As described above, the server computing system 130 may store or include one or more machine-learned augmented reality rendering models 140. For example, the models 140 may be or otherwise include various machine-learned models. Exemplary machine-learned models include neural networks or other multi-layer nonlinear models. Examples of neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, convolutional neural networks, etc. Exemplary models 140 are described with reference to FIGS. 5-9.

[0050] User computing device 102 and / or server computing system 130 can train models 120 and / or 140 through interaction with a training computing system 150 that is communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0051] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operably connected processors. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing device 150 to perform operations. In some implementations, the training computing device 150 includes or is implemented by one or more server computing devices.

[0052] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored on the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation of error. For example, a loss function may be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent may be used to iteratively update the parameters over multiple training iterations.

[0053] In some implementations, performing backpropagation of the error may include performing truncated backpropagation over time. The model trainer 160 may perform a number of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0054] In particular, model trainer 160 can train augmented reality models 120 and / or 140 based on a set of training data 162. Training data 162 can include, for example, shaders built by a third party using a software development kit, where the third party received the software development kit from the facilitating computing device or server computing system 130. The third party may have generated the shaders and data assets by building and testing an augmented reality experience using the software development kit.

[0055] In some implementations, if the user consents, the training examples can be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process can be referred to as personalizing the model.

[0056] In some implementations, the perception and rendering models of the augmented reality model may be trained using template images, which may be images from a corpus of template images that are provided to the user as pre-rendered images, or in some implementations, training template images may be a separate set of template images used only for training.

[0057] Model trainer 160 includes computer logic utilized to provide desired functionality. Model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some implementations, model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as RAM, a hard disk, or an optical or magnetic medium.

[0058] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications on network 180 may be transmitted over any kind of wired and / or wireless connection using a variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0059] The machine learning models described herein may be used in a variety of tasks, applications, and / or use cases.

[0060] In some implementations, an input to a machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a modification of the image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning model may process the image data to generate an upscaled image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0061] In some implementations, the input to the machine learning model of the present disclosure may be text or natural language data. The machine learning model can process the text or natural language data to generate an output. As an example, the machine learning model can process the natural language data to generate a language-encoded output. As another example, the machine learning model can process the text or natural language data to generate a potential text-embedding output. As another example, the machine learning model can process the text or natural language data to generate a translation output. As another example, the machine learning model can process the text or natural language data to generate a classification output. As another example, the machine learning model can process the text or natural language data to generate a text segmentation output. As another example, the machine learning model can process the text or natural language data to generate a semantic intent output. As another example, the machine learning model can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As another example, the machine learning model can process the text or natural language data to generate a predicted output.

[0062] In some implementations, input to a machine learning model of the present disclosure may be latent encoding data (e.g., a latent spatial representation of the input, etc.). The machine learning model may process the latent encoding data to generate an output. As an example, the machine learning model may process the latent encoding data to generate a recognition output. As another example, the machine learning model may process the latent encoding data to generate a reconstruction output. As another example, the machine learning model may process the latent encoding data to generate a search output. As another example, the machine learning model may process the latent encoding data to generate a reclustering output. As another example, the machine learning model may process the latent encoding data to generate a prediction output.

[0063] In some implementations, an input to a machine learning model of the present disclosure may be sensor data. The machine learning model may process the sensor data to generate an output. As an example, the machine learning model may process the sensor data to generate a recognition output. As another example, the machine learning model may process the sensor data to generate a prediction output. As another example, the machine learning model may process the sensor data to generate a classification output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a visualization output. As another example, the machine learning model may process the sensor data to generate a diagnostic output. As another example, the machine learning model may process the sensor data to generate a detection output.

[0064] In some cases, the machine learning model may be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). In another example, the input includes visual data (e.g., one or more images or videos), the output comprises compressed visual data, and the task is a visual data compression task. In another example, the task may comprise generating an embedding of the input data (e.g., the visual data).

[0065] In some cases, the input includes visual data and the task is a computer vision task. In some cases, the input includes pixel data of one or more images and the task is an image processing task. For example, the image processing task is image classification and the output is a set of scores, each score corresponding to a different object class and representing the likelihood that one or more images depict an object belonging to that object class. The image processing task may be object detection and the image processing output identifies one or more regions in one or more images and, for each region, the likelihood that the region represents an object of interest. As another example, the image processing task may be image segmentation and the image processing output defines, for each pixel in one or more images, a respective likelihood for each category in a set of predetermined categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task is depth estimation and the image processing output defines, for each pixel in one or more images, a respective depth value. As another example, the image processing task is motion estimation and the network input includes multiple images and the image processing output defines, for each pixel in one of the input images, the motion of the scene depicted by the pixel between images in the network input.

[0066] 1A illustrates one exemplary computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, a user computing device 102 can include a model trainer 160 and a training dataset 162. In such implementations, the model 120 can be both trained and used locally on the user computing device 102. In some such implementations, the user computing device 102 can implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0067] 1B illustrates a block diagram of an exemplary computing device 10 that may perform in accordance with an exemplary embodiment of the present disclosure. Computing device 10 may be a user computing device or a server computing device.

[0068] The computing device 10 includes multiple applications (e.g., applications 1 through N). Each application includes its own machine learning library and machine learning model. For example, each application may include a machine learning model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0069] 1B , each application can communicate with many other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0070] 1C illustrates a block diagram of an exemplary computing device 50 for performing operations according to an exemplary embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0071] Computing device 50 includes multiple applications (e.g., applications 1 through N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0072] The central intelligence layer includes multiple machine learning models. For example, as shown in FIG. 1C , a respective machine learning model (e.g., model) may be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included within or implemented by the operating system of computing device 50.

[0073] The central intelligence layer can communicate with a central device data layer, which can be a central repository of data for computing device 50. As shown in FIG. 1C , the central device data layer can communicate with many other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0074] Example Model Configuration 2 illustrates a block diagram of an example implementation 200 according to an example embodiment of the present disclosure. In some implementations, the example implementation 200 includes a search result list 204 describing results of a search query, and provides a pre-rendered image 208 as a result of receiving the search result list 204, the pre-rendered image 208 being a pre-rendered augmented reality image including a resulting product rendering of one or more results 206 from the search result list 204. Accordingly, in some implementations, the example implementation 200 may include a search query input area 202 operable to receive a search query input.

[0075] The exemplary implementation of FIG. 2 illustrates a search engine web service 210. A search engine input area 202 can capture a search query, which can include one or more search terms. The platform can capture and process the search query to provide a search result list 204. The one or more search terms can relate to products for purchase. In these implementations, the search result list 204 can be one or more products related to the search terms. In some implementations, one or more results 206 from the search result list 204 can include one or more pre-rendered images 208 associated with the result 206. In some implementations, the pre-rendered image 208 can include a rendering of the corresponding result 206. In some implementations, the user can further select a virtual live try-on augmented reality experience.

[0076] 3 illustrates a block diagram of an example implementation 300 according to an example embodiment of the present disclosure. The example implementation 300 is similar to the example implementation 200 of FIG. 2, except that the example implementation 300 further illustrates an example result.

[0077] FIG. 3 illustrates an example implementation 300 as a furniture try-on experience. For example, to generate the pre-rendered image shown in the example implementation 300, a product 302 can be rendered within a template environment. In this example, a loveseat 304, a television 306, and a rug 308 are part of a template image of an example room. This template image can be one of multiple template images that can be processed to generate multiple pre-rendered images. In this example, the multiple pre-rendered images can include a rendering of the product 302 within each corresponding template image. The template images can have various sizes, themes, and configurations. For example, this example pre-rendered image includes a television 306 across from a loveseat 304, with a rug 308 in between. In some implementations, the platform can allow a user to view the rendered product in various locations within the template image.

[0078] 4 illustrates a block diagram of an example implementation 400 according to an example embodiment of the present disclosure. The example implementation 400 is similar to the example implementation 200 of FIG. 2, except that the example implementation 400 further illustrates an example implementation in a mobile application.

[0079] In the exemplary implementation shown in FIG. 4 , the platform is accessed through a user interface within a mobile application on a mobile device 402. A user can use the mobile device to access the mobile application, where the user can store preferences and access a library of pre-rendered images. For example, a user may prefer a particular living room template image 408 of their choosing. The user may have selected that template image 408 as the closest match to the user's living room. The template image 408 may include a couch 404 and a similar lamp 406 found in the user's home. The template image 408, along with other template images, may have been processed to generate various pre-rendered images of various furniture and decor products. Using the mobile application, the platform can provide various decor or furniture products rendered within the user's preferred template image 408 to assist the user's shopping experience. For example, a user may use the mobile application to see how a particular rug would look under a couch 404 and lamp 406 similar to the setup in their living room.

[0080] 5 illustrates a block diagram of an example platform 500 according to an example embodiment of the present disclosure. In some implementations, the platform 500 is trained to receive requests and preferences 506 describing user-specific preferences, and to provide output data as a result of receiving the preferences 506, including pre-rendered images of augmented reality renderings on template images associated with the user-specific preferences 506. Accordingly, in some implementations, the platform 500 may include a directory 510 operable to store the template images, augmented reality data assets, and pre-rendered images.

[0081] The example platform of Figure 5 includes a directory 510 of pre-rendered images and a user 502. The user can have settings 504 selected by the user, which can include preferences 506. In some implementations, the preferences 506 can include selected preferences associated with a template image. The preferences 506 can be used to determine which pre-rendered image from multiple pre-rendered images can be provided to the user when a request is made. For example, a preference associated with a first template image can cause the platform to provide a pre-rendered image that includes a product rendered in the first template image.

[0082] The directory 510 can store template images, augmented reality assets, and pre-rendered images. The template images can be for various environments, including, but not limited to, rooms, gardens, driveways, and faces. The augmented reality assets can be used to render various objects and products, including, but not limited to, furniture, decorations, plants, electronics, automobiles, and makeup. In some implementations, the pre-rendered images can include products rendered within the template images, in which case the products can be rendered based on the augmented reality assets.

[0083] In some implementations, the platform can store pre-rendered images of a variety of different products in various environments. For example, dataset 1 can include various pre-rendered images 512 of lamps in various living rooms. Dataset 2 can include multiple pre-rendered images 514 of lipstick on a face. Dataset 3 can include multiple pre-rendered images 516 of trees in a garden. In some implementations, a user can use a user interface provided by a web service, mobile application, or kiosk to access the pre-rendered images to assist with shopping.

[0084] 9 illustrates a block diagram of an exemplary augmentation platform 900, according to an exemplary embodiment of the present disclosure. In some implementations, the augmentation platform 900 is trained to receive a set of input data describing a user request and to provide output data including a pre-rendered rendering of a template image as a result of receiving the input data. Thus, in some implementations, the augmentation platform 900 may include an augmented reality pre-rendering platform 920 that is operable to interact with a user device and enable an augmented reality pre-rendering experience.

[0085] The augmented reality pre-rendering platform 920 shown in FIG. 9 includes a user interface 922 for enabling user interaction, a template library 924 for processing, a rendering engine 926 for processing the template images, and a pre-rendered library 928 for storing pre-rendered images generated by processing the template images.

[0086] In some implementations, the augmented reality pre-rendering platform 920 can receive user preferences 912 from the user computing device 910. The user preferences 912 can be used to determine which of a plurality of pre-rendered images to provide to the user computing device 910 from the pre-rendered library 928.

[0087] In some implementations, the augmented reality pre-rendering platform 920 can provide an option for an augmented reality live try-on experience, which can involve processing user media data or user camera feed 914 by a rendering engine 926 to generate renderings of user-provided data.

[0088] Exemplary Methods 6 shows a flowchart diagram of an exemplary method for performing in accordance with an exemplary embodiment of the present disclosure. While FIG. 6 shows steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. Various steps of method 600 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0089] At 602, the computing system may obtain a plurality of template images. In some implementations, the template images may include room images, face images, or garden images. The plurality of template images may include images of various environments. For example, the plurality of template images may include a variety of different room sizes, configurations, themes, lighting, or colors.

[0090] At 604, the computing system can process a plurality of template images with an augmented reality rendering model. The augmented reality rendering model can include object tracking and rendering. Processing the template image with the augmented reality rendering model can generate a plurality of pre-rendered images. The plurality of pre-rendered images can include augmented reality renderings rendered within each of the plurality of template images. The augmented reality renderings can describe a product. The augmented reality rendering model can be based at least in part on data assets provided by a third party, where the data assets are generated by the third party by building an augmented reality rendering experience that renders the product in images and videos. For example, the product being rendered can be furniture (e.g., a couch, a chair, a table, etc.). The plurality of pre-rendered images can include a particular couch rendered within a plurality of different template images depicting a variety of different rooms having different sizes, colors, themes, and configurations. In another example, the product can be a makeup product (e.g., lipstick, mascara, foundation, eyeliner, eyeshadow, etc.). In this implementation, the plurality of pre-rendered images can include renderings of a makeup product such as lipstick. In this implementation, lipstick can be rendered onto multiple template face images.

[0091] In some implementations, processing with the augmented reality rendering model may include processing a plurality of template images with a perceptual model to generate a mesh and a segmentation mask, and then processing the mesh, segmentation mask, and template images with the rendering model to generate a plurality of pre-rendered images.

[0092] At 606, the computing system can receive a request and preferences for result images. The request can include search terms entered into a search engine. In some implementations, the preferences can include pre-selected templates.

[0093] At 608, the computing system can provide a pre-rendered result. The pre-rendered result can be based at least in part on the request and the preferences. The pre-rendered result can be a pre-rendered image from a plurality of pre-rendered images. In some implementations, the pre-rendered result can be a pre-rendered image that matches the preferences, where the preferences can include a template selected by the user from a plurality of template images.

[0094] In some implementations, the computing system can provide an augmented reality experience. The augmented reality experience can be provided upon receiving a selection to proceed to the augmented reality experience.

[0095] 7 shows a flowchart diagram of an exemplary method for performing in accordance with an exemplary embodiment of the present disclosure. While FIG. 7 shows steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. Various steps of method 700 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0096] At 702, a computing system can obtain an augmented reality asset. The augmented reality asset can include digitized parameters.

[0097] At 704, a computing system can obtain a plurality of template images.

[0098] At 706, the computing system can process the plurality of template images with the augmented reality model to generate a plurality of pre-rendered images. The plurality of pre-rendered images can be generated based at least in part on the digitization parameters. In some implementations, the computing system can generate a plurality of template models based at least in part on the plurality of template images. The augmented reality model can then process the plurality of template models to generate a plurality of pre-rendered images.

[0099] In some implementations, the computing system can include receiving input to modify a template model of the plurality of template models, where the template model can be modified based at least in part on the template. In some implementations, the template model can be provided for display.

[0100] The computing system may store the plurality of pre-rendered images at 708. In some implementations, the plurality of pre-rendered images may be stored on a server.

[0101] The computing system can provide the stored pre-rendered images to a user. In some implementations, the computing system can receive a search query including one or more search terms, the one or more search terms related to a product. The computing system can then provide search results, the search results including a pre-rendered image from the plurality of pre-rendered images. In some implementations, the pre-rendered image can include a rendering of the product.

[0102] In some implementations, the computing system may provide a link to a real-time augmented reality experience.

[0103] 8 shows a flowchart diagram of an exemplary method for performing in accordance with an exemplary embodiment of the present disclosure. While FIG. 8 shows steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. Various steps of method 800 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0104] At 802, a computing system can obtain an augmented reality asset. The augmented reality asset can include digitized parameters.

[0105] At 804, a computing system can obtain a plurality of template images.

[0106] At 806, the computing system can process the plurality of template images with the augmented reality model to generate a plurality of pre-rendered images. The augmented reality model can generate the plurality of pre-rendered images based at least in part on the digitization parameters.

[0107] At 808, the computing system can store the multiple pre-rendered images.

[0108] At 810, the computing system may receive a search query. The search query may include one or more search terms, where the one or more search terms are related to a product.

[0109] At 812, the computing system may provide search results. The search results may include pre-rendered images from a plurality of pre-rendered images retrieved from a server, and the pre-rendered images may include renderings of the products.

[0110] Additional Disclosures The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions performed on and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of configurations, combinations, and divisions of tasks and functionality among components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications may be implemented in a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0111] While the subject matter of the present invention has been described in detail with reference to various specific exemplary embodiments thereof, each example is provided by way of illustration and not by way of limitation of the disclosure. Those skilled in the art, upon understanding the foregoing, will be able to readily create modifications, variations, and equivalents to such embodiments. Accordingly, the present disclosure is not intended to preclude inclusion of such modifications, variations, and / or additions to the subject matter as would be readily apparent to one skilled in the art. For example, features illustrated or described as part of one embodiment can be used with another embodiment to yield yet another embodiment. Accordingly, the present disclosure is intended to cover such modifications, variations, and equivalents. [Explanation of symbols]

[0112] 50 computing devices 102 User Computing Devices 112 processors 114 memory 116 Data 118 Command 120 Augmented Reality Rendering Models 120 Machine Learning Models 122 User Input Components 130 Server Computing System 132 processors 134 memory 136 Data 138 Command 140 Augmented Reality Rendering Models 150 Training Computing System 152 processors 154 memory 156 Data 158 Command 160 Model Trainer 162 training data 162 training datasets 180 Network 200 Implementation Example 202 Search query input area, search engine input area 204 search results list 206 results 208 pre-rendered images 210 Search Engine Web Services 300 Implementation Example 302 products 304 Loveseat 306 Television 308 Rug 400 Implementation Example 402 Mobile Devices 404 Couch 406 Lamp 408 Living Room Template Images 500 Platform 502 users 504 Settings 506 Preference 510 Directory 512 Various pre-rendered images of lamps in various living rooms 514 Multiple pre-rendered images of lipstick on face 516 Multiple pre-rendered images of garden trees 600 ways 700 methods 800 ways 900 Extended Platform 910 User Computing Device 912 User Preferences 914 User Camera Feeds 920 Augmented Reality Pre-rendering Platform 922 User Interface 924 Template Library 926 rendering engine 928 Pre-rendered Library

Claims

1. 1. A computing system comprising: one or more processors; one or more computer-readable storage media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations; and and wherein the operation comprises: storing a plurality of pre-rendered images associated with the product in a directory of pre-rendered images, the plurality of pre-rendered images being generated by processing a plurality of template images with an augmented reality rendering model, the plurality of pre-rendered images including a rendering of the product within each corresponding template image, the plurality of template images being selected from a corpus of images that are a representative group of template images; receiving a search query including one or more search terms, the one or more search terms related to a product; obtaining a plurality of pre-rendered images comprising renderings of products based on the search query, the plurality of pre-rendered images being obtained from a directory of pre-rendered images; generating a carousel interface, the carousel interface including the plurality of pre-rendered images including renderings of products based on the search query; providing the carousel interface for display; a computing system including:

2. The computing system of claim 1 , wherein the plurality of pre-rendered images describes a particular product rendered using a plurality of different template images.

3. The computing system of claim 1 , wherein the plurality of pre-rendered images describe a plurality of different environments.

4. The computing system of claim 1 , wherein the carousel interface further comprises an augmented reality virtual try-on panel.

5. The computing system of claim 1 , wherein the products include at least one of cosmetics, household goods, clothing, paint colors, automobiles, or electronic devices.

6. 10. The computing system of claim 1, wherein the plurality of pre-rendered images are obtained from a pre-rendered library that includes pre-rendered images for a plurality of different products using various template images.

7. The computing system of claim 1 , wherein the plurality of pre-rendered images are obtained based at least in part on user preferences, and the carousel interface includes personalized previews.

8. The operation is receiving preferences, the preferences being associated with a preselected template; 2. The computing system of claim 1, wherein the plurality of pre-rendered images includes pre-rendered augmented images, the pre-rendered augmented images including the product rendered within the pre-selected template.

9. The operation is acquiring one or more augmented reality assets; acquiring one or more template images; processing the one or more template images with the augmented reality rendering model to generate a plurality of pre-rendered images; storing the plurality of pre-rendered images; The computing system of claim 1 further comprising:

10. The computing system of claim 9 , wherein the augmented reality assets include data describing the product.

11. 1. A computer-implemented method for providing pre-rendered augmented images, comprising: a computing system including one or more processors storing a plurality of pre-rendered images associated with a product in a pre-rendered image directory, the plurality of pre-rendered images being generated by processing a plurality of template images with an augmented reality rendering model, the plurality of pre-rendered images including a rendering of the product in each corresponding template image, the plurality of template images being selected from a corpus of images that are a representative group of template images; receiving, by the computing system, a search query including one or more search terms, the one or more search terms related to a product; the computing system obtaining a plurality of pre-rendered images including renderings of products based on the search query, the plurality of pre-rendered images being obtained from a directory of pre-rendered images; generating, by the computing system, a carousel interface, the carousel interface including the plurality of pre-rendered images including renderings of products based on the search query; providing, by the computing system, a search results page, the search results page including the carousel interface for display; 11. A computer-implemented method comprising:

12. The computer-implemented method of claim 11 , wherein the search results page includes one or more search results provided for display adjacent to the carousel interface.

13. The computer-implemented method of claim 11 , wherein the search results page includes one or more search results related to the product.

14. The computer-implemented method of claim 11 , wherein the plurality of pre-rendered images comprises an augmented reality rendering of the product rendered on a plurality of different surfaces.

15. The computer-implemented method of claim 11 , wherein the plurality of pre-rendered images comprises an augmented reality rendering of the product rendered in a plurality of different environments.

16. One or more computer-readable storage media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations including: receiving a search query including one or more search terms, the one or more search terms related to a product; retrieving a plurality of pre-rendered images comprising renderings of products based on the search query, the plurality of pre-rendered images being retrieved from a directory of pre-rendered images, the plurality of pre-rendered images comprising: acquiring an augmented reality asset, the augmented reality asset including digitized parameters; obtaining a plurality of template images, the plurality of template images being selected from a corpus of images that are representative groups of template images; processing the plurality of template images with the augmented reality model to generate the plurality of pre-rendered images based at least in part on the digitization parameters and including a respective corresponding rendering of the product in each template image; and obtaining the generated generating a carousel interface, the carousel interface including the plurality of pre-rendered images including renderings of products based on the search query; providing the carousel interface for display; [0023] 1. One or more computer-readable storage media, including:

17. The one or more computer-readable storage media of claim 16 , wherein the augmented reality assets include data describing the product.

18. 17. The one or more computer-readable storage media of claim 16, wherein the search query includes a preference associated with a pre-selected template image, and the plurality of template images includes the pre-selected template image.

19. 17. The one or more computer-readable storage media of claim 16, wherein the carousel interface is provided for display via a user interface provided by an augmented reality pre-rendering platform.

20. 17. The one or more computer-readable storage media of claim 16, wherein the search query is received via a user interface of a mobile application, and the carousel interface is provided for display via the user interface of the mobile application.

Images