Computing platform for facilitating augmented reality experiences with third party assets

Abstract

To provide a computing platform for facilitating augmented reality experiences with third party assets.SOLUTION: Systems and methods for data asset acquisition and obfuscation are helpful for retrieving augmented reality rendering data assets from third parties. Sending a software development kit and returning data assets can confirm that the data assets are compatible with an augmented reality rendering experience in a user interface. The data acquisition system with the obfuscation also ensures that a code generated by third parties is stripped of semantics and has reduced readability.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] Related Applications This application claims priority to and the benefit of U.S. Non-provisional Patent Application No. 17 / 144,002, filed January 7, 2021. U.S. Non-provisional Patent Application No. 17 / 144,002 is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to acquiring data assets and, more particularly, to acquiring data assets from third parties to provide an augmented reality experience to a user. [Background technology]

[0003] Augmented reality (AR) can refer to the creation and execution of interactive experiences of real-world environments in which objects present in the real world are augmented with computer-generated perceptual information. As an example, an AR experience can include augmenting a scene captured by a user's camera by inserting virtual objects into the scene and / or modifying the appearance of real-world objects included in the scene.

[0004] Certain entities may offer users the ability to participate in AR experiences related to objects manufactured or distributed by that entity (e.g., an augmented reality "try-on"). However, most entities only offer these AR experiences on their own websites. Due to limited accessibility to augmented reality product "try-ons," consumers may need to navigate from one website to another to participate in different AR experiences for different objects (e.g., to try on different products and compare products). This issue may cause consumers to expend more computing resources to navigate between different websites to facilitate participation in different AR sessions.

[0005] Additionally, web-based implementations of augmented reality experiences can result in lower frame rates and less realistic rendering compared to augmented reality experiences provided in native applications.

[0006] Another issue with participating in different augmented reality experiences is that different AR applications may be coded in different coding languages, which may pose security issues, which may be of particular concern if the application processes images or video of the user's face. Summary of the Invention [Means for solving the problem]

[0007] Aspects and advantages of embodiments of the disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0008] One exemplary aspect of the present disclosure is directed to a computer-implemented method for providing a third-party data asset to a client. The method can include transmitting a software development kit by a computing device. The software development kit can include templates for building one or more rendering effect shaders. The method can include receiving the data asset by the computing device. In some implementations, the data asset can include one or more rendering effect shaders built using the software development kit. The method can include storing the data asset by the computing device and providing an augmented reality rendering experience by the computing device. In some implementations, the augmented reality rendering can be based at least in part on the data asset.

[0009] Another exemplary aspect of the present disclosure is directed to a computing system. The computing system may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include receiving a software development kit comprising a template for generating an augmented reality experience. The operations may include receiving one or more inputs for modifying the template. In some implementations, modifying the template may generate a product-specific augmented reality experience. In some implementations, the operations may include generating a data asset based at least in part on the one or more inputs and transmitting the data asset to a second computing device.

[0010] Another exemplary aspect of the present disclosure is directed to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations. The operations can include transmitting a software development kit by the computing device. In some implementations, the software development kit can include templates for building one or more rendering effect shaders. The operations can include receiving data assets by the computing device. The data assets can include one or more rendering effect shaders built using the software development kit. The operations can include storing the data assets by the computing device and providing an augmented reality rendering experience by the computing device. In some implementations, the augmented reality rendering can be based at least in part on the data assets.

[0011] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0012] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain the relevant principles.

[0013] Detailed descriptions of embodiments, directed to those skilled in the art, are set forth herein with reference to the accompanying drawings. [Brief description of the drawings]

[0014] [Figure 1A] FIG. 1 is a block diagram of an exemplary computing system for performing data asset acquisition in accordance with an exemplary embodiment of the present disclosure. [Figure 1B] FIG. 1 is a block diagram of an exemplary computing system that performs acquisition and obfuscation of data assets in accordance with an exemplary embodiment of the present disclosure. [Figure 1C] FIG. 2 is a block diagram of an example computing device that performs acquisition and obfuscation of data assets in accordance with an example embodiment of the present disclosure. [Figure 1D] FIG. 2 is a block diagram of an example computing device that performs acquisition and obfuscation of data assets in accordance with an example embodiment of the present disclosure. [Diagram 2] 1 is a block diagram of an exemplary data asset acquisition system according to an exemplary embodiment of the present disclosure. [Diagram 3] FIG. 2 is a block diagram of an example obfuscation according to an example embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram of an exemplary data asset acquisition system in accordance with an exemplary embodiment of the present disclosure. [Diagram 5] FIG. 2 is a block diagram of an exemplary augmented reality rendering experience model, according to an exemplary embodiment of the present disclosure. [Figure 6]FIG. 2 is a flowchart diagram of an example method for performing data asset acquisition, according to an example embodiment of the present disclosure. [Figure 7] FIG. 2 is a flowchart diagram of an exemplary method for performing data asset generation, according to an exemplary embodiment of the present disclosure. [Figure 8] FIG. 2 is a flowchart diagram of an exemplary method for performing acquisition and obfuscation of a data asset, according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Reference numbers that are repeated among the drawings are intended to identify like features in various implementations.

[0016] Overview In general, the present disclosure is directed to a platform that facilitates collection of data assets from third parties to provide an augmented reality rendering experience to a user. For example, the platform can provide an interface to a third party to build and / or submit rendering effects to be provided to a user. The platform can provide an augmented reality rendering experience to a user, where the user can input user data and receive augmented user data as output. Additionally, according to another exemplary aspect, the platform can perform various obfuscation techniques on received third-party data assets to protect the third-party code.

[0017] Thus, exemplary aspects of the present disclosure can provide a system for acquiring third-party data assets for augmented reality rendering and obfuscating the acquired code to protect confidential information of the third party. In some implementations, the system and method can include sending a software development kit to the third party. The software development kit can include templates for building rendering effect shaders. As an example, the AR effects can be organized into a shared template and multiple presets per product. The templates can include shaders and other resources that are common to all products in a particular family (e.g., all lipsticks from a particular vendor). The presets can include parameters corresponding to those shaders and can include uniform values ​​and textures. The third party can provide the data assets to the platform (e.g., packaged in an SDK). As an example, the data assets can include one or more rendering effect shaders built using the software development kit. The received data assets can be stored. The data assets can then be used to facilitate an augmented reality rendering experience, and the data assets are useful in generating an augmented reality try-on experience.

[0018] The systems and methods for data asset acquisition may involve one or more systems or devices. The first computing device may be a server, a facilitating computing device, or an intermediary computing device. The second computing device may be a third-party computing device. The third party may be a video game company, a product manufacturer, or a product brand. The first computing device and the second computing device may exchange data to generate an augmented reality rendering experience for the user. The augmented reality rendering experience may include rendering an augmented reality view that includes one or more products or items. The products may be cosmetics (e.g., lipstick, eye shadow, etc.), furniture or other household items (e.g., electronics, cookware, glassware, ornaments, plants, etc.), clothing, paint colors, automobiles, various electronic devices, or other items.

[0019] The acquisition of the data assets may include the first computing device sending a software development kit to the second computing device. The software development kit may include templates for constructing rendering effect shaders. The software development kit may include example effects, tools for constructing the rendering effect, and a preview mode useful for constructing the augmented reality rendering. The second computing device may be used to construct the rendering effect, and once the rendering effect is constructed, the second computing device may export the constructed rendering effect data to a renderable compressed file (e.g., a .ZIP file) that may include the data assets necessary to recreate the rendering effect. The data assets may then be sent to the first computing device. Upon receiving the data assets, the first computing device may store the data assets for use in an augmented reality rendering experience provided to a user. The provided augmented reality rendering experience may be provided to a user, where the user may input user data for processing, and the output may be augmented user data including the rendering effect constructed on the second computing device. The user data may be image data or video data captured by the user device. In some implementations, the user data may be a live camera feed.

[0020] In some implementations, one or more rendering effect shaders can include data describing rendering capabilities. One or more rendering effect shaders can be used to render rendering effects for an augmented reality experience. For example, rendering lipstick on a face using augmented reality can include utilizing a lipstick shader, textures, uniforms, and filtering shaders. In some implementations, textures and uniforms can be input to the shaders to aid in generating the rendering effect. In some implementations, the data assets can include rendering effect shaders. In some implementations, the rendering effect shaders can be processed by a rendering calculator along with user data to generate a rendered image.

[0021] In some implementations, the received data assets can be used to facilitate the generation of an augmented reality rendering. The mesh model may be able to process the input data to generate a mesh and a segmentation mask. The augmented model may then receive the mesh and segmentation mask and use the data assets to determine where and how to render a particular rendering. For example, the data assets can include anchor landmark data. Thus, the data assets can be used to determine where the rendering needs to occur. In some implementations, the data assets can include shader data. The shader data may describe the color of a lipstick or the color of an eye shadow to assist in generating a makeup augmented reality rendering for a user. Additionally, the shader data may describe the color, glossiness, opacity, and / or various other characteristics of the product.

[0022] The augmented reality rendering experience can include a rendering calculator that generates a rendering based at least in part on the data assets. In some implementations, the augmented reality rendering experience can include a rendering engine, and the rendering engine can include the rendering calculator.

[0023] The augmented reality rendering experience may include a perception subgraph and a rendering subgraph. The perception subgraph may be uniform across the system. The perception subgraph may be used with a variety of different rendering subgraphs. The rendering subgraph may be constructed by a third party to generate the rendering effect provided to the user. The rendering subgraph may be constructed and then used by the augmented reality rendering experience platform that stores the perception subgraph. The rendering subgraph may vary depending on the rendering effect and the third party. In some implementations, a single perception subgraph may be used with multiple rendering subgraphs to render multiple renderings in an augmented user image or video. For example, a photo or video of a user's face may be processed to generate an augmented reality rendering of lipstick, eyeshadow, and mascara on the face. The processing may include a single perception subgraph, but may include rendering subgraphs for each product (i.e., lipstick, eyeshadow, and mascara).

[0024] In some implementations, the data assets can include product data describing products sold by a third party. In some implementations, the systems and methods disclosed herein can be used to compile product augmented reality rendering experiences for a retailer to allow consumers to virtually try on different products from a variety of different brands or providers. The retailer can be an online retailer, where the consumer can virtually try on products in the comfort of their home. In some implementations, the retailer can be a physical store retailer, where the augmented reality experience can be enabled through a mobile application or computing device located within the store. In some implementations, the systems and methods disclosed herein can enable augmented reality rendering experiences within a search engine application where a consumer may search for a product brand, product type, product color, etc., and the search engine can provide results based on the search query, where the results include an option to try on the results determined using augmented reality.

[0025] In some implementations, the received data assets enable the generation of various augmented reality experiences. For example, the data assets may include data assets generated by manufacturers, distributors, sellers, etc. of furniture or other household items to aid in rendering one or more of their products in a home. In this implementation, a third party may modify and tweak the software development kit to enable the generation of augmented reality experiences of furniture or other household items. If the third party is satisfied with the generated experience, the data assets may be exported and sent to a facilitation system / platform for storage. The facilitation system may use the data assets to enable users to access the augmented reality rendering experience.

[0026] The augmented reality rendering can be generated by receiving user data, processing the user data with an encoder model to generate a user mesh, and processing the user mesh with an augmented model to generate the augmented reality rendering. In some implementations, the mesh can be a polygonal mesh. In some implementations, the augmented model can include a shader based at least in part on the data assets. In some implementations, the augmented model can generate the augmented reality rendering by rendering a polygonal mesh using a custom shader and overlaying it on a camera feed image.

[0027] In some implementations, the system and method can include a third-party computing device that imports and modifies the software development kit. The method can include receiving a software development kit configured to be modified to generate an augmented reality experience. Additionally, the method can include receiving one or more inputs to modify the software development kit, where modifying the software development kit generates a product-specific augmented reality experience. The method can include generating data assets based at least in part on the one or more inputs and transmitting the data assets to a second computing device.

[0028] In some implementations, the third-party computing device may test the augmented reality experience for fine-tuning. Testing the augmented reality experience may include comparing a rendered image with a pre-rendered reference ("golden") image. Alternatively, in some implementations, testing may include obtaining training data, processing the training data with the augmented reality experience to generate the augmented reality media, and comparing the augmented reality media with the reference data. One or more parameters of the data assets may be adjusted (e.g., automatically using a machine learning algorithm) based at least in part on the comparison (e.g., based on a loss function that compares the augmented reality media with the reference data).

[0029] In some implementations, the system and method may process the received data asset to obfuscate the code. The obfuscation may include removing whitespace from the code, removing one or more comments from the code, and renaming terms and symbols in the code. The one or more comments may be removed as they may contain text that describes the semantics of the code. Renaming one or more terms in the code of the data asset may include uniform renaming across files. The uniform renaming process may include indexing terms to provide references for future uniform renaming. In some implementations, the renaming may include using a hash function. The hash may be indexed in a global registry or index table. New symbols may be actively added to the index as they are received. The renamed terms and symbols may be deterministic terms. In some implementations, the system may rename symbols such as function names that are shared across multiple files.

[0030] Renaming may involve parsing the shader code and isolating specific declarations to replace. Removing whitespace may involve removing code organization to remove some logic contained in formatted code in a certain way. Obfuscation may involve removing semantic content that may contain shader information that a third party does not want exposed (e.g., sensitive information that may be in OpenGL code, GLSL, or other native sources).

[0031] In some implementations, the platform can be used to generate a directory of product renderings. A directory of augmented reality renderings can be implemented to provide renderings for advertisements, video web applications, or mobile apps. In some implementations, shared code can be integrated into the directory as a library dependency.

[0032] Receiving or ingesting data assets can occur through a variety of methods. In some implementations, ingest can occur through an export function built into the platform. In other implementations, data assets can be delivered through emailing a single effect template parameterized by shader uniforms defined in a single top-level file. For example, for lipstick rendering, a third party can send shader uniform values ​​for each lipstick product they want to provide to the user. The system and method can combine the effect template and the shader uniforms per product to generate an AR effect (e.g., a beauty effect) per product. In some implementations, data assets can be ingested through an internet-based data feed instead of email. Data assets can be ingested individually or in bulk.

[0033] In some implementations, software development kits can be configured for different product types. For example, a software development kit for building data assets for rendering lipsticks can include different templates and presets than a software development kit for building data assets for rendering sofas. Additionally, beauty effects can be organized into shared "templates" and multiple presets per product to help third parties build data assets for rendering specific products. Templates can include shaders and other resources that are common to all products in a particular family (e.g., all lipsticks from vendor A). Presets can include parameters corresponding to those shaders and can include uniform values ​​and textures.

[0034] In some implementations, the systems and methods disclosed herein can be implemented as a native application that can provide the client with an augmented reality rendering experience that includes renderings of third-party products selected by the client.

[0035] The systems and methods disclosed herein are also applicable to other technologies, including mixed reality. In some implementations, a software development kit can be used by a third party to build interactive renderings. For example, a furniture brand may build different rendering effects for different reclining chairs and different extendable tables. The rendering effect shaders can be transferred to an augmented reality rendering experience platform to provide a mixed reality rendering experience where a user can render the furniture in their home. The user can then interact with the rendering to see the rendering of the furniture in a default position and in another position. Thus, the user can test how a reclining chair might fit in their home in an upright position and a reclined position. The user can use the mixed reality experience to determine if an extended table will fit in a given room.

[0036] Additionally, the data asset acquisition and obfuscation platform can be applied to a variety of other platforms to generate supported user experiences. The data acquisition platform can be utilized to create supported applications, build embedded features, and create widgets.

[0037] Further, in some implementations, the system and method may be used as a visual compatibility calculator. For example, the system and method may be used to ensure that a particular product or part will fit a desired space or location. The system and method may be used to virtually test the dimensions / size of a product using virtual reality. A third party may provide a data asset that includes data describing the measurements of a product. The data asset may then be used to provide a user with an augmented reality rendering experience in which the product is rendered according to the measurements provided by the third party. This aspect allows a consumer to "try on" the product to visualize the space the product will occupy.

[0038] The system and method of the present disclosure provide many technical effects and advantages. As an example, the system and method can receive parameters for augmented reality from a third party to allow a client to virtually view a product in an application. The system and method can also be used to ensure the security of data provided by a third party. Furthermore, the system and method enable centralized collection of augmented reality data sets of virtual try-on clothes from various vendors to allow a user to try on products from multiple vendors without having to move from website to website or from proprietary application to proprietary application.

[0039] Additionally, in some implementations, the disclosed systems and methods can be implemented in native applications, which can provide higher frame rates and more realistic rendering compared to web application alternatives.

[0040] Another technical advantage of the systems and methods of the present disclosure is the ability to obfuscate code (e.g., shader code), which may be clustered without comments or symbols that would reveal sensitive information contained in the originally provided source code. The systems and methods may also obfuscate received data assets, allowing third parties to provide data without revealing certain sensitive information.

[0041] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the drawings.

[0042] Exemplary Devices and Systems 1A illustrates a block diagram of an example computing system 80 for performing data asset acquisition in accordance with an example embodiment of the present disclosure. The system 80 includes a client computing system 20, an augmented reality platform 30, and a third-party computing system.

[0043] 1A , the augmented reality platform 30 can communicate with a third-party computing system 60 to generate a third-party augmented reality (AR) asset library 40. The third-party AR asset library 40 can be leveraged to provide an augmented reality (AR) experience to the client computing system 20 via the client interface 32.

[0044] For example, the augmented reality platform 30 can provide a software development kit (SDK) with templates to the third-party computing system 60. The third-party computing system 60 can use the SDK with templates to build an augmented reality rendering effect that describes a product sold by the third party. The completed rendering effect can be returned to the augmented reality platform 30 as a completed data asset. Each completed data asset can be stored in the third-party AR asset library 40 of the augmented reality platform 30. The product assets stored in the third-party AR asset library 40 can be from a single third party or from multiple third parties. For example, product 1 assets 42, product 2 assets 44, through product N assets 46 can be from a single third party computing system and include various products provided by the third parties. Alternatively, product 1 assets 42 and product 2 assets 44 can be provided by different third parties and include data assets describing different products of different brands.

[0045] The augmented reality platform 30 can take in client data via the client interface 32 for processing by the rendering engine 34 to provide an AR experience to the client computing system 20. The rendering engine 34 can process the client data with a perception model and an augmented model. The perception model can output one or more meshes and one or more segmentation masks that can be input into the augmented model. The augmented model can process the client data, the one or more meshes, and the one or more segmentation masks to output an augmented reality client image or video, which can be transmitted to the client computing system 20 via the client interface 32.

[0046] In some implementations, the AR experience can include the client computing system 20 sending a selection to the augmented reality platform 30 to indicate a desired product to "try on." The augmented reality platform can use stored data assets from a third-party AR asset library 40 to render the product in an image or video provided by the client by parameterizing an augmented model of the rendering engine 34 with the data assets. For example, the client computing system 20 can use the client interface 32 to select a product 1 to virtually try on. The product 1 asset 42 can be provided to the rendering engine 34 along with a set of client data. The rendering engine 34 can process the set of client data and the product 1 asset 42 to generate an image or video of the product 1 in the image or video provided by the client computing system 20.

[0047] 1B illustrates a block diagram of an exemplary computing system 100 for performing acquisition and obfuscation of data assets in accordance with an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0048] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0049] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or multiple processors operably connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0050] In some implementations, the user computing device 102 can store or include one or more augmented reality rendering models 120. For example, the augmented reality rendering models 120 can be or otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models including nonlinear and / or linear models. The neural networks can include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. An exemplary augmented reality rendering model 120 is described with reference to FIGS. 2 and 4.

[0051] In some implementations, one or more augmented reality rendering models 120 may include data assets received from the training computing system 150 over the network 180, stored in the user computing device memory 114, and then used or implemented by one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single augmented reality rendering model 120 (e.g., to perform parallel rendering of effects).

[0052] More specifically, the server computing system 130 and the training computing system 150 can exchange data to generate data assets that can enable the augmented reality rendering model to process image or video data and output augmented image data or augmented video data.

[0053] Additionally or alternatively, one or more augmented reality rendering models 140 may be included in or stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the augmented reality rendering models 140 may be implemented by the server computing system 130 as part of a web service (e.g., a "live try-on" service for makeup, clothing, electronics, automobiles, or furniture or other household items). Thus, one or more models 120 may be stored and implemented on the user computing device 102 and / or one or more models 140 may be stored and implemented on the server computing system 130.

[0054] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that senses the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component functions to implement a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0055] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0056] In some implementations, the server computing system 130 includes or is implemented by one or more server computing devices. When the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0057] As described above, the server computing system 130 may store or include one or more machine-learned augmented reality rendering models 140. For example, the models 140 may be or otherwise include various machine-learned models. Exemplary machine-learned models include neural networks or other multi-layer nonlinear models. Examples of neural networks include feed-forward neural networks, deep neural networks, recurrent neural networks, convolutional neural networks, etc. Exemplary models 140 are described with reference to FIGS. 2 and 4.

[0058] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 through interaction with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0059] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing device 150 to perform operations. In some implementations, the training computing device 150 includes or is implemented by one or more server computing devices.

[0060] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored on the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation of error. For example, a loss function may be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent may be used to iteratively update the parameters over multiple training iterations.

[0061] In some implementations, performing backpropagation of the error may include performing truncated backpropagation over time. The model trainer 160 may perform a number of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0062] In particular, model trainer 160 can train the augmented reality rendering models 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, shaders built by a third party using a software development kit, where the third party received the software development kit from the facilitating computing device or server computing system 130. The third party may have generated the shaders and data assets by building and testing an augmented reality experience using the software development kit.

[0063] In some implementations, if the user consents, the training examples can be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process can be referred to as personalizing the model.

[0064] Model trainer 160 includes computer logic utilized to provide desired functionality. Model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some implementations, model trainer 160 includes program files stored in a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored in RAM, a hard disk, or a tangible computer-readable storage medium, such as an optical or magnetic medium.

[0065] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications on network 180 may be transmitted over any kind of wired and / or wireless connections using a variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0066] The machine learning models described herein may be used in a variety of tasks, applications, and / or use cases.

[0067] In some implementations, an input to the machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of the image data, latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., modification of the image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning model may process the image data to generate an upscaled image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0068] In some implementations, the input to the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a potential text embedding output. As another example, the machine learning model may process the text or natural language data to generate a translation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0069] In some implementations, the input to the machine learning model of the present disclosure may be latent encoding data (e.g., a latent spatial representation of the input, etc.). The machine learning model may process the latent encoding data to generate an output. As an example, the machine learning model may process the latent encoding data to generate a recognition output. As another example, the machine learning model may process the latent encoding data to generate a reconstruction output. As another example, the machine learning model may process the latent encoding data to generate a search output. As another example, the machine learning model may process the latent encoding data to generate a reclustering output. As another example, the machine learning model may process the latent encoding data to generate a prediction output.

[0070] In some implementations, the input to the machine learning model of the present disclosure may be sensor data. The machine learning model may process the sensor data to generate an output. As an example, the machine learning model may process the sensor data to generate a recognition output. As another example, the machine learning model may process the sensor data to generate a prediction output. As another example, the machine learning model may process the sensor data to generate a classification output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a visualization output. As another example, the machine learning model may process the sensor data to generate a diagnostic output. As another example, the machine learning model may process the sensor data to generate a detection output.

[0071] In some cases, the machine learning model may be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). In another example, the input includes visual data (e.g., one or more images or videos) and the output comprises compressed visual data, and the task is a visual data compression task. In another example, the task may comprise generating an embedding of the input data (e.g., visual data).

[0072] In some cases, the input includes visual data and the task is a computer vision task. In some cases, the input includes pixel data of one or more images and the task is an image processing task. For example, the image processing task is image classification and the output is a set of scores, each score corresponding to a different object class and representing the likelihood that the one or more images depict an object belonging to that object class. The image processing task may be object detection and the image processing output identifies one or more regions in the one or more images and, for each region, the likelihood that the region represents an object of interest. As another example, the image processing task may be image segmentation and the image processing output defines, for each pixel in the one or more images, a respective likelihood for each category in a set of predefined categories. For example, the set of categories can be foreground and background. As another example, the set of categories can be object classes. As another example, the image processing task is depth estimation and the image processing output defines, for each pixel in the one or more images, a respective depth value. As another example, the image processing task is motion estimation and the network input includes multiple images and the image processing output defines, for each pixel in one of the input images, the motion of the scene depicted in the pixel between images in the network input.

[0073] 1B illustrates one exemplary computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device 102 can include a model trainer 160 and a training dataset 162. In such implementations, the model 120 can be both trained and used locally at the user computing device 102. In some such implementations, the user computing device 102 can implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0074] 1C illustrates a block diagram of an exemplary computing device 10 for performing operations according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0075] The computing device 10 includes multiple applications (e.g., applications 1 through N). Each application includes its own machine learning library and machine learning model. For example, each application may include a machine learning model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0076] 1C, each application may communicate with many other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application may communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0077] 1D illustrates a block diagram of an exemplary computing device 50 for performing operations according to an exemplary embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0078] Computing device 50 includes multiple applications (e.g., applications 1 through N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and models stored therein) using an API (e.g., a common API across all applications).

[0079] The central intelligence layer includes multiple machine learning models. For example, as shown in FIG. 1D, each machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included within or implemented by an operating system of the computing device 50.

[0080] The central intelligence layer can communicate with a central device data layer, which can be a central repository of data for the computing device 50. As shown in FIG. 1D, the central device data layer can communicate with many other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0081] Example Model Configuration In some implementations, the systems and methods can be used as a rendering pipeline. The pipeline can include a software development kit (SDK) that can include all the tools necessary to build a renderable compressed file (e.g., a ZIP file). In some implementations, the software development kit that includes the compressed file (e.g., a ZIP file) can be tested with viewers on various platforms. Source assets generated during construction using the software development kit can be used to enhance still images or videos.

[0082] The software development kit can be sent to third parties who can then build and preview the inbound rendering pipeline. The pipeline can be used for a variety of purposes, including, but not limited to, trying on beauty products using a computer or mobile device.

[0083] The software development kit may include scripts with associated binaries, a set of preview tools, documentation, and a set of sample effects. The scripts and associated binaries may be used to compile the product effect source into a renderable compressed file. The set of preview tools may be used to visualize the rendering results. Additionally, the preview tools may provide a unified interface for editing and previewing, or quick switching between the two interfaces. The documentation may include raw HTML or other formatted documentation for review to aid in the build process. The set of sample effects may help third parties understand the software development kit while providing a baseline.

[0084] The software development kit can be designed to be self-contained, easily portable, capable of running on stock hardware, fast, and secure. The software development kit can be designed to run without dependencies, allowing third parties or other creators to build renderings using the software development kit without relying on other applications. The backend can mirror the backend used by the system that interfaces with the consumer. Additionally, the ease of porting allows third parties to use existing shaders with minimal or no modifications. Additionally, the software development kit can be designed to run on a variety of operating systems without the need to install software outside of the software development kit. In some implementations, the software development kit can include functionality that allows opening up the render graph and GPU shaders for customization. This interface can eliminate the need to implement third-party GPU code on the system that interfaces with the consumer, maintaining user security. The software development kit can use a rendering calculator to convert incoming data to outgoing data.

[0085] The generation of an inward-facing augmented reality effect may include two components. The first component may include a perception component. The perception component may calculate and output pixel coordinates of landmarks (e.g., lips on a face) in an image. The second component may include a rendering component. The rendering component may include rendering the augmented reality effect on the original received frame using the calculated landmarks to facilitate location identification. The result may then be output.

[0086] The augmented reality rendering graph can be split into a third party subgraph and a facilitator subgraph. The facilitator subgraph can be a perception subgraph, and the third party subgraph can be a rendering subgraph that can be edited by both the third party and the facilitator. The separation allows for modification of the perception subgraph by the facilitator without affecting the rendering subgraph. Furthermore, this separation allows for a single perception subgraph to be used even when multiple augmented reality effects are being rendered. Thus, multiple rendering subgraphs can be overlaid on one perception subgraph to generate multiple renderings using the computations of a single perception model process.

[0087] The perception subgraph and the rendering subgraph can be aggregated to generate a complete graph or augmented reality media.

[0088] A third-party effect source can include a rendering sub-graph file and one or more directories of assets used by the rendering calculator within the sub-graph.

[0089] In some implementations, the software development kit may include a perceptual model. The software development kit may call bundled binaries to convert each type of source file into a processed format, allowing a directory structure of inputs to be saved into the generated renderable compressed file while producing errors made along the way. Asset files such as graph protos, shaders, and blueprints may be included in the software development kit or may reference other files that are part of the effect source. Textures may be converted to WebP format based on a per-effect image_conversion.txt file that customizes the conversion.

[0090] In some implementations, product effects built with the software development kit can include GLSL shaders that can be provided to a user's device for runtime compilation. These shaders can represent valuable intellectual property from third-party technology providers. The systems and methods disclosed herein can obfuscate the shaders to help protect the information. Obfuscation can include removing comments and syntactically redundant whitespace, and then systematically renaming most unreserved symbols to obscure semantics. In some implementations, the obfuscated symbols can be uniformly obfuscated across files. Obfuscation can occur after inspecting and testing the original shaders obtained from the third-party partner.

[0091] The systems and methods can be applied to a variety of augmented reality renderings, including, but not limited to, makeup renderings, furniture renderings, apparel renderings, video game renderings, and architectural construction renderings.

[0092] 2 illustrates a block diagram of an exemplary data asset acquisition system 200 according to an exemplary embodiment of the present disclosure. In some implementations, the data asset acquisition system 200 is trained to transmit a software development kit 212 for building an augmented reality rendering experience and to receive templates and presets 228 from a third party as a result of transmitting the software development kit 212. Thus, in some implementations, the data acquisition system 200 can include a facilitator 210, a third party 220, and a user 250.

[0093] In particular, Figure 2 illustrates a system for data asset acquisition for use in generating an augmented reality rendering experience. A facilitator 210 can be a facilitation system for compiling data assets for augmented reality rendering by communicating with a third party 220. The facilitator 210 can be a server, a web application, or a facilitation computing system. Once the facilitator 210 receives the data assets, the facilitation system can provide an augmented reality rendering experience to a user 250.

[0094] The acquisition of data assets may include a facilitator-built software development kit (SDK) 212 that is sent to a third party 220. The third party may use the software development kit 222 to build the rendering experience. The software development kit 222 may compile the data assets 224 and may enable a rendering preview 226 of the generated rendering experience.

[0095] The third parties 220 may use the rendering preview to determine what, if any, source modifications 230 need to be made to their data asset templates and presets 228. Once constructed, tested, and fine-tuned, the third parties can submit the data assets including the templates and presets 228 and third party metadata 232 to the facilitation system. The facilitation system 210 can capture the effects 214 and store them for later providing an augmented reality rendering experience that can include the rendering effects constructed by the third parties 220. The facilitator can capture user data 252 from the user 250 and output augmented user data including the rendering effects included in the augmented user data.

[0096] 3 illustrates a block diagram of an example obfuscation 300 according to an example embodiment of the present disclosure. The obfuscation 300 can be included in the data asset acquisition system 200 of FIG. 2 to obfuscate a data asset.

[0097] More specifically, Figure 3 illustrates a three-pronged approach to obfuscation. In this implementation, an original code 302 can be obfuscated using an obfuscation system 304 to generate an obfuscated code 306. In some implementations, the original code 302 can be a data asset code for augmented reality rendering.

[0098] Additionally, in this implementation, the obfuscation system 304 can include renaming 308 symbols or terms, removing 310 whitespace, and removing 312 semantic comments.

[0099] Renaming 308 can include creating an index of symbols and terms from the original code and replacing the original symbols and terms with the assigned symbols and terms. In some implementations, renaming 308 can include utilizing a hash function. Renaming can be uniform across various files.

[0100] Removing whitespace 310 may include removing indentations, blank lines, and line breaks. Removing whitespace can obscure the logical form of the code, making the code harder to read.

[0101] Removing comments 312 may include removing one or more comments related to code semantics. The obfuscation system may process the code and determine whether a comment is a semantic comment. If the comment helps explain the semantics of the code, it may be removed.

[0102] These three components can reduce code readability while at the same time concealing third-party symbols and terminology.

[0103] 4 illustrates a block diagram of an exemplary data asset acquisition system 400 according to an exemplary embodiment of the present disclosure. The data asset acquisition system 400 is similar to the data asset acquisition system 200 of FIG. 2, except that the data asset acquisition system 400 may be specifically configured for beauty effect rendering.

[0104] In this implementation, the cosmetic effect 402 can include makeup rendering. The cosmetic effect can include various datasets. The datasets can include a dataset of render entities 404 including blueprints, GLSL shaders 406, textures, and geometry files. The cosmetic effect 402 can further include rendering subgraphs, shader uniforms, and image transforms. These datasets can be input into a software development kit 408 to build an augmented reality rendering experience. The software development kit 408 allows the builder to preview and test the newly built experience. In particular, the GLSL shaders 406 of the cosmetic effect can be converted into a shader pipeline within the software development kit. The built augmented reality rendering experience can run parallel to a previously existing rendering experience of the cosmetic effect 402 outside of the software development kit 408. Once built, the software development kit can generate a renderable zip file 410 for use in recreating the augmented reality rendering experience on another device.

[0105] 5 illustrates a block diagram of an exemplary augmented reality rendering experience model 500 according to an exemplary embodiment of the present disclosure. The augmented reality rendering experience model 500 is similar to the data asset acquisition system 200 of FIG. 2, except that the augmented reality rendering experience model 500 further includes processing the camera feed with a mesh model and the augmented model with an augmented model using generated data assets.

[0106] In particular, FIG. 5 illustrates an example perception subgraph 504 and an example rendering subgraph 508 used to process the camera feed 502 to generate a rendered image 510. The perception subgraph 504 can process the camera feed 502 with a mesh model 506 to generate a mesh and a segmentation mask. The mesh and segmentation mask can be processed by a rendering calculator 512. The rendering calculator 512 can be included in the rendering subgraph 508 and can be affected by shaders. The mesh, segmentation mask, and the camera feed 502 can be processed by the rendering calculator 512 to generate a rendered image. The rendered image can include an augmented reality rendering. The rendering can be a rendering generated by a third party using a software development kit. Additionally, in some implementations, the rendering can be a makeup rendering, the mesh model can be a face tracker, and the shaders can include a lipstick shader, a texture shader, a uniform shader, and / or a filtering shader.

[0107] Exemplary Methods 6 illustrates a flow chart diagram of an exemplary method for performing in accordance with an exemplary embodiment of the present disclosure. Although FIG. 6 illustrates steps performed in a particular order for purposes of illustration and discussion, the method of the present disclosure is not limited to the particular order or arrangement illustrated. Various steps of method 600 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0108] At 602, the computing system can transmit a software development kit. In some implementations, the software development kit can include templates for building one or more rendering effect shaders. The software development kit can include binaries associated with scripts for compiling source effects into renderable files. The software development kit can further include a preview tool for configuration third parties to preview the rendering build during construction for tweaking and testing. In some implementations, the software development kit can include documentation and sample effects to assist third parties on how to build and tweak effects using the software development kit.

[0109] At 604, the computing system can receive a data asset. In some implementations, the data asset can include one or more rendering effect shaders built using a software development kit. The data asset can be received in the form of a renderable file (e.g., a .ZIP file). The data asset can include shader data for rendering a product or item sold or offered by a third party. The product can be cosmetics (e.g., lipstick, eye shadow, etc.), furniture or other household items (e.g., electronics, cookware, glassware, ornaments, plants, etc.), clothing, paint colors, automobiles, or other items.

[0110] The computing system may store the data asset at 606. The data asset may be stored on a server, a user computing device, or a facilitator computing device.

[0111] At 608, the computing system can provide an augmented reality rendering experience. In some implementations, the augmented reality rendering can be based at least in part on the data assets. The augmented reality rendering can include a product rendering of a product sold by a third party.

[0112] 7 shows a flow chart diagram of an exemplary method for performing in accordance with an exemplary embodiment of the present disclosure. Although FIG. 7 shows steps performed in a particular order for purposes of illustration and discussion, the method of the present disclosure is not limited to the particular order or arrangement shown. Various steps of method 700 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0113] At 702, a computing system can receive a software development kit. The software development kit can include templates for generating an augmented reality experience. The software development kit can be transmitted by a facilitating computing system or an intermediary system to compile a rendering experience for provision to a user.

[0114] At 704, the computing system can receive input to modify the software development kit. Modifying the software development kit can modify the template and generate a product-specific augmented reality experience. For example, the software development kit can be constructed and configured to generate a makeup rendering effect. The makeup rendering effect can be implemented in an augmented reality rendering experience where a user can "try on" various lipstick colors or types offered using a personal computing device.

[0115] At 706, the computing system can generate a data asset. In some implementations, the data asset can be generated based at least in part on the received input. The data asset can be converted into a compressed file that can be rendered for transmission.

[0116] At 708, the computing system can transmit the data assets to a second computing device. The second computing device may be a sender of a software development kit. In some implementations, the second computing device may be a facilitating computing device or an intermediary computing device that interacts with a user to provide the user with an augmented reality rendering experience.

[0117] 8 shows a flow chart diagram of an exemplary method for performing in accordance with an exemplary embodiment of the present disclosure. Although FIG. 8 shows steps performed in a particular order for purposes of illustration and discussion, the method of the present disclosure is not limited to the particular order or arrangement shown. Various steps of method 800 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0118] At 802, the computing system can transmit a software development kit. In some implementations, the software development kit can include templates for building one or more rendering effect shaders. The base application can be an augmented reality rendering application with example effects and instructions for building new rendering effects or transforming previously built rendering effects.

[0119] At 804, the computing system can receive a data asset. In some implementations, the data asset can include one or more rendering effect shaders constructed using a software development kit. The data asset can be received as a renderable file.

[0120] At 806, the computing system can obfuscate the data asset. Obfuscating the data asset can include removing whitespace from the code, removing comments about the semantics of the code, and renaming symbols and terms. Obfuscation can reduce the readability of the code and remove sensitive information that may be contained in terms or comments. Obfuscation is useful to protect third-party builders.

[0121] At 808, the computing system can store the data assets. The data assets can be stored locally or via a server. The data assets can be stored so that they are readily accessible for use in a web or mobile application.

[0122] At 810, the computing system can provide an augmented reality rendering experience. In some implementations, the augmented reality rendering can be based at least in part on the data assets. The augmented reality rendering experience can include providing an experience that augments a user's images and videos to include rendering effects built by a third party. The rendering effects can be part of a video game or a "live try-on" experience. Additionally, the augmented reality rendering experience can be provided via a web application, a mobile application, or an in-store kiosk.

[0123] Additional Disclosures The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken on such systems and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of configurations, combinations, and divisions of tasks and functions among components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications may be implemented in a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0124] Although the subject matter of the present invention has been described in detail with respect to various specific exemplary embodiments thereof, each example is provided for illustrative purposes and not to limit the present disclosure. Those skilled in the art, upon understanding the foregoing, can easily create modifications, variations, and equivalents to such embodiments. Thus, the present disclosure is not intended to preclude inclusion of such modifications, variations, and / or additions to the subject matter as would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment can be used with another embodiment to obtain yet another embodiment. Thus, the present disclosure is intended to cover such modifications, variations, and equivalents. [Explanation of symbols]

[0125] 10. Computing Devices 20 Client Computing Systems 30 Augmented Reality Platform 32 Client Interface 34 Rendering Engine 40 Third-Party Augmented Reality (AR) Asset Library 42 Product 1 Asset 44 Product 2 Assets 46 Product N Assets 50 Computing Devices 60 Third Party Computing Systems 80 Computing Systems 102 User Computing Devices 112 processors 114 Memory 116 Data 118 Command 120 Augmented Reality Rendering Models 120 Machine Learning Models 122 User Input Components 130 Server Computing System 132 processors 134 Memory 136 Data 138 Command 140 Augmented Reality Rendering Models 140 Machine Learning Models 150 Training Computing System 152 processors 154 Memory 156 Data 158 Command 160 Model Trainer 162 training data 162 training dataset 180 Network 200 Data Asset Acquisition System 210 Facilitator 210 Facilitation System 212 Software Development Kit 214 Effects 220 Third Party 222 Software Development Kit 224 Data Assets 226 Rendering Preview 228 Templates and Presets 230 Source correction 232 Third Party Metadata 250 users 252 User Data 300 Obfuscated 302 Original Code 304 Obfuscation System 306 Obfuscated Code 308 Change name 310 Whitespace Removal 312 Delete comment 400 Data Asset Acquisition System 402 Beauty Effects 404 Render Entity 406 GLSL Shaders 408 Software Development Kit 410 renderable zip files 500 Augmented Reality Rendering Experience Models 502 Camera Feed 504 Perceptual Subgraph 506 Mesh Model 508 Rendering Subgraphs 510 Rendered Images 512 Rendering Calculator 600 ways 700 methods 800 ways

Claims

1. A computing system comprising: one or more processors; and one or more computer-readable storage media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations including: receiving a software development kit, the software development kit comprising templates for generating an augmented reality experience utilizing one or more rendering effects shaders, the software development kit comprising one or more per-product presets, the templates and the one or more per-product presets being associated with products of a particular product class, the one or more per-product presets comprising one or more parameters associated with uniform values ​​and textures for the particular product class; receiving one or more inputs to modify the template, where modifying the software development kit generates a product-specific augmented reality experience; generating a data asset based at least in part on the one or more inputs, the data asset being associated with one or more products of the particular product class; transmitting the data assets to a second computing system; Equipped with Computing system.

2. The computing system of claim 1, wherein the software development kit includes an integrated interface for editing and previewing augmented reality experiences.

3. The operation The computing system of claim 1 , further comprising providing the product-specific augmented reality experience for display.

4. The operation processing the data assets to generate obfuscated code; transmitting the obfuscated code to the second computing system; The computing system of claim 1 further comprising:

5. The method of claim 1, further comprising: testing the augmented reality experience; and testing the augmented reality experience. Obtaining training data; processing the training data with the augmented reality experience to generate an augmented reality media; evaluating a loss function based at least in part on a comparison of the augmented reality media to ground truth data; adjusting one or more parameters based at least in part on the loss function; The computing system of claim 1 , comprising:

6. the data asset comprises code; The operation, determining that one or more comments in the code comprise text that describes the semantics of the code; removing said one or more comments comprising text describing the semantics of the code; The computing system of claim 1 further comprising:

7. The computing system of claim 1, wherein the software development kit is received from the second computing system.

8. The computing system of claim 1, wherein generating the data asset based at least in part on the one or more inputs comprises generating a renderable compressed file comprising the data asset associated with rendering a product-specific rendering effect.

9. The computing system of claim 1, wherein the product-specific augmented reality experience comprises an augmented reality try-on experience that renders the one or more products in user image data.

10. 2. The computing system of claim 1, wherein the data assets comprise a rendering subgraph associated with the one or more products, the rendering subgraph being processed together with a perception subgraph to provide the product-specific augmented reality experience.

11. A computer-implemented method comprising: receiving, by a computing system comprising one or more processors, a software development kit comprising templates for generating an augmented reality experience utilizing one or more rendering effects shaders, the software development kit comprising one or more per-product presets, the templates and the one or more per-product presets being associated with products of a particular product class, the one or more per-product presets comprising one or more parameters associated with uniform values ​​and textures for the particular product class; receiving, by the computing system, one or more inputs for modifying the template, wherein modifying the software development kit generates a product-specific augmented reality experience; generating, by the computing system, a data asset based at least in part on the one or more inputs, the data asset being associated with one or more products of the particular product class; transmitting, by the computing system, the data asset to a second computing device; Equipped with method.

12. The method of claim 1, further comprising the step of obfuscating, by the computing system, the data asset, the step of obfuscating comprising: removing, by the computing system, whitespace from the code of the data asset; renaming, by the computing system, one or more terms in the code; The method of claim 11 comprising:

13. The method of claim 12, wherein the step of renaming one or more terms in the code by the computing system comprises uniform renaming across an entire file.

14. The method of claim 11 , wherein the data assets comprise anchor landmark data associated with a location for rendering a rendering effect.

15. The method of claim 11, wherein the data assets comprise shader data associated with a product color and a product opacity.

16. One or more computer-readable storage media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations including: receiving a software development kit, the software development kit comprising templates for generating an augmented reality experience utilizing one or more rendering effects shaders, the software development kit comprising one or more per-product presets, the templates and the one or more per-product presets being associated with products of a particular product class, the one or more per-product presets comprising one or more parameters associated with uniform values ​​and textures for the particular product class; receiving one or more inputs to modify the template, where modifying the software development kit generates a product-specific augmented reality experience; generating a data asset based at least in part on the one or more inputs, the data asset being associated with one or more products of the particular product class; transmitting the data asset to a second computing device; Equipped with One or more computer-readable storage media.

17. The method of claim 15, further comprising: processing the data assets to generate obfuscated code; 20. The one or more computer-readable storage media of claim 16, wherein generating the obfuscated code comprises renaming one or more terms in code for the data asset.

18. One or more computer-readable storage media as described in claim 17, comprising a uniform renaming process, wherein renaming the one or more terms in the code of the data asset comprises indexing the renamed terms to provide reference for future uniform renaming.

19. One or more computer-readable storage media as described in claim 17, wherein renaming the one or more terms in the code of the data asset comprises renaming using a hash function, and the generated hash is indexed in a global registry.

20. One or more computer-readable storage media as described in claim 16, wherein the data asset comprises one or more size parameters associated with a visual compatibility calculator.

Images