Product as key for context-based image generation

Abstract

The invention relates to the use of a product as a key for context-based image generation. In some embodiments, a method of using a customized product as a key for generating a context-based image, the method comprising: receiving, by a visualization service of a computer collaboration system, a graphical representation of a customized digital product; determining one or more regions for customizing the region; for one of the regions: determining a set of phrases specific to the customized digital product and a customization process for customizing the customized digital product; generating a composite image generation request based on the phrase set; transmitting the request to an AI-based image generator configured to generate a composite image; causing the AI-based generator to: generate a customized composite image based on the request and the image repository; wherein the customized composite image is not a copy of any image from the image repository; transmitting the customized synthetic image to a computer collaboration system; including the image in the region; a graphical visualization of the customized digital product is caused to be displayed in the area along with the customized composite image.

Description

Technical Field

[0001] One technical area of ​​the present disclosure is digital image processing. Another technical area is computer-implemented techniques for generating digital composite images based on requests created from various information derived from customized digital content, customized features, user profiles, etc. Another technical area is computer-implemented techniques for merging the generated digital composite images with customized content using digital image processing and accurately and in near real time producing depictions of customizable items mixed with the generated composite images. Background Art

[0002] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

[0003] As digital computing power grows and manufacturing processes improve, the manufacturing of goods is shifting to producing goods based on individual customized requests received from customers' computers. For example, this may include fulfilling online orders received from customers who use their computer systems to customize depictions of generic goods and order individual customized goods online.

[0004] Many systems on the market are configured to provide users with the opportunity to order products with customized attributes. For example, for custom-made frame products such as photographs, digital images, artwork, and other products, the system can provide users with the opportunity to order frames with custom sizes and colors.

[0005] Customizing a product with many customizable parameters can be quite challenging. The choice of customization values ​​may have an impact on the appearance of the final customized product and how the final customized product is rendered. Therefore, the system usually provides the function of displaying a depiction (e.g., a composite view) of the customized product to help users intuitively understand their customized product before ordering the product.

[0006] However, current systems typically provide stock images as backgrounds for displaying customized products. These images are typically prepared in advance and selected in a predictable and unchangeable manner for displaying customized products. Furthermore, in some cases, the images in the image library rarely relate to the actual products that the user has customized and ordered. Furthermore, these images rarely allow for an enhanced user experience when the user customizes and examines the digital depiction of the customized product displayed on a computer-based display device.

[0007] Therefore, there is a need to generate and render synthetic backgrounds and visual components for effectively visualizing customized products and in sync with the product customization process. Such synthetic backgrounds and visual components need to be generated based on specific information about the customized product and the user performing the customization. Furthermore, such synthetic images need to be generated instantly and quickly enough to avoid any delays in rendering the customized product overlaid with the synthetic background image, etc. Frame rendering time in real-time rendering applications should be fast, as latency appears to be a key barrier to enabling user interaction with the application. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1A is a block diagram illustrating an example environment for using a product as a key for context-based image generation;

[0009] Figure 1B are examples of options and processing byproducts;

[0010] Figure 1C are examples of options and processing byproducts;

[0011] Figure 1D is an example of customization;

[0012] Figure 1E is an example of customization;

[0013] Figure 1F is an example of customization;

[0014] Figure 1G is an example of customization;

[0015] Figure 1H is an example of customization;

[0016] Fig. 1I is an example of getting the base image;

[0017] Figure 2A is an example of utilizing AI-based image generation methods;

[0018] Figure 2B is an example of merging AI input and corresponding AI output with custom images;

[0019] Figure 3A is a block diagram illustrating an example environment for designing and manufacturing products;

[0020] Figure 3B is a block diagram illustrating an example of a role-based collaboration platform;

[0021] Figure 4 is a flow chart depicting an example process for using a product as a key for context-based image generation;

[0022] Figure 5is a block diagram illustrating a computer system in which the techniques described herein may be implemented. DETAILED DESCRIPTION

[0023] In the following description, for the purpose of explanation, numerous specific details are set forth to provide a thorough understanding of the methods described herein. However, it is apparent that the present invention can be practiced without these specific details. In other cases, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the present invention.

[0024] Various embodiments are described herein according to the following outline:

[0025] 1. General Overview

[0026] 2. Generate synthetic background and image

[0027] 2.1. Computing Environment

[0028] 2.2. Synthesizing Background and Image

[0029] 2.3. Stable diffusion

[0030] 3. Select content for the build request

[0031] 4. Sample Request

[0032] 5. Base Image

[0033] 6. Use AI to generate synthetic backgrounds / images

[0034] 6.1. Request

[0035] 6.2. AI-based Image Generator

[0036] 6.3. Synthetic Images

[0037] 7. Example Manufacturing System

[0038] 7.1. Description of the structure of the authoring tool

[0039] 7.2. Design assets

[0040] 7.3. Asset definition data

[0041] 7.4. User Equipment

[0042] 7.5. Client Application

[0043] 7.6. Front-end Server

[0044] 7.7. Core Services

[0045] 7.8. Example Manufacturing System

[0046] 7.9. Example Manufacturing Process

[0047] 8. Sample Product Collaboration Platform

[0048] 9. Example Flowchart

[0049] 10. Implementation Mechanism

[0050] 1. General Overview

[0051] The present disclosure proposes a technical solution to a technical problem, and any interpretation that interprets the present disclosure or claims as covering any judicial exception to patent eligibility (such as abstract concepts, mental processes, methods for organizing human activities, or mathematical algorithms) is not supported in the present disclosure and is erroneous.

[0052] Artificial intelligence (AI) and AI-based tools have great potential in image generation and customization. For example, AI-based tools can generate synthetic images and provide a good user experience and rich new features for tasks that were originally difficult and mundane.

[0053] Among other things, an AI-based approach can be used to generate images for product customization, for example, in online shopping digital customization. This may include guiding a user through the process of creating and generating instant backgrounds and other images to enhance the craft of customizing a digital product. The instantly generated backgrounds and other images can be customized in a variety of ways, including matching the backgrounds and images to the customized product in a sense, as described later.

[0054] A unique aspect of the presented approach is that the instantly generated backgrounds and images are truly instantly generated and are generated or synthesized based on images from a large publicly available image library and using provided keywords, terms, descriptions, etc. The instantly generated images are not actual images from the image library. Instead, the instantly generated images are uniquely generated and synthesized based on images stored in the resource library. The algorithms used to generate / synthesize AI-based instant images are described later.

[0055] Suppose the customized product is a Christmas tree decoration. The corresponding instantly generated background can be a Christmas-related scene that matches the customized product in terms of occasion, color scheme, etc. The instantly generated background can be synthesized from a large number of Christmas-related images stored in the resource library. This AI-generated background can be included in the 3D visualization / depictment of the customized product (e.g., Christmas tree decoration). When the product is further customized by changing the product color, etc., a new background can be generated on the fly to match the color scheme of the customized product, etc.

[0056] According to another example, assuming that the customized product is a handbag, the customization includes designing a color application insert to be applied to the front of the handbag. A textual description of the desired color application can be used to generate an instant image to be used as the insert for the handbag. For example, if the description of the desired application includes the words "Flowers in Paris in Spring", then the corresponding instant generated image can be generated from a large number of pictures depicting flowers in Paris. The generated composite image can depict flowers blooming in a Paris park, etc.

[0057] In some cases, multiple instantly generated backgrounds and images may be generated for a customized product. These backgrounds and images may be superimposed or otherwise included in the depiction of the customized product. Including such synthetic backgrounds and images allows for improved quality of product customization and improves the quality of product display.

[0058] In some cases, the background and image generated on the fly are part of the product depiction. The depiction may also include information of a UV-marked area, which is defined by field marks, number marks, a yellow grid, an inverted yellow grid, a grid deformation, a mask, or other means. For example, if the depiction includes a UV-marked area within a custom product, the image generated on the fly may be applied over the area defined by the UV mark.

[0059] According to another example, if the depiction includes a masked area, the background generated on the fly can be applied to the masked area. The mask can be used to delineate an area, for example, in which the background can be replaced with an AI-based generated image.

[0060] 2. Generate synthetic background and image

[0061] 2.1. Computing Environment

[0062] Figure 1A is a block diagram showing an example environment 100A for using a product as a key to generate context-based images. FIG. 1 , other figures, and all descriptions and claims in this disclosure are intended to propose, disclose, and advocate a technical system and multiple technical methods in which specially programmed computers use special-purpose distributed computer system designs to perform unprecedented functions to provide practical applications of computing technology for the development, verification, and deployment problems of machine learning models. In this way, the present disclosure proposes a technical solution to the technical problem, and any interpretation that interprets the present disclosure or claims as covering any judicial exception to patent eligibility (such as abstract concepts, mental processes, methods of organizing human activities, or mathematical algorithms) is unsupported and erroneous in the present disclosure.

[0063] Figure 1AThe example of environment 11 described in is configured to use products as keys for context-based image generation. One of the elements of environment 11 is a computer collaboration system 100, which will be described in detail later. For clarity of description, computer collaboration system 100 includes visualization service 100A, which in turn includes request generator 152, image requester 158, image overlay 160, and rendering framework 130 (described in detail later). All of these components will be described in detail later.

[0064] The collaboration system 100 may also include other components, which will be described by way of example later.

[0065] In some embodiments, the environment 11 also includes a database 172A (described later) for storing key-value pairs, a database 172B (described later) for storing user profiles, and one or more other databases 172N for storing additional information used by the environment 11 and / or the components of the computer collaboration system 100.

[0066] Environment 11 may also include a database 174A for storing, for example, billions of digital images from publicly available digital images. These images may be downloaded from public resources, databases, and other repositories. For example, these images may be downloaded from the Internet and may represent a portion of a DALL-E system (described later).

[0067] Images stored in database 174A or other storage space included in environment 11 may be used by AI-based image generator 180. AI-based image generator 180 may be an AI-based application configured to, for example, implement a stable diffusion method for generating or synthesizing images in response to receiving a text query.

[0068] 2.2. Synthesizing Background and Image

[0069] The AI-based techniques used in the present method may include various generative AI techniques (GAI). An example of such techniques includes stable diffusion (SD). In this case, SD can be used to generate a digital image context of a customized product without changing the customized product itself or changing the key that traces back to the product database version of the customized product. An example of SD technology is briefly described below.

[0070] Assume that the customized product is a handbag, and the handbag is displayed on an online shopping website so that the user can digitally customize the handbag and order it. The customized product (i.e., the handbag here) can be customized in many ways. For example, the front of the handbag can be customized. Figure 1B An example of such customization is described.

[0071] Figure 1B is an example of options and byproducts of processing. Assume that the base image of the front of a customized product (e.g., a handbag) is plain cloth. However, in order to make the handbag appear more attractive on the display screen of an online shopping website, it may be necessary to replace the plain cloth front of the handbag with an attractive picture that corresponds in some sense to the user's interests, the user's profile, etc. In addition, in order to make the handbag appear more attractive on the display screen, it is better to replace the monochrome background with an attractive background, such as images of a beach, traveling along the beach, playing on the beach, etc. These images and backgrounds can be generated on the fly using SD, etc., which will be introduced later.

[0072] To achieve product customization, the computer collaboration system 100 (such as Figure 1A and described later) can use one or more graphical representations of the product, as well as several masks for selecting customizable areas in the product description, which are filled with AI-based synthetic images, and several maps for implementing lighting models, light reflections, etc. These representations, masks, and maps are generally used to make the depiction of the product more attractive, more realistic, and more relevant to the user's interests.

[0073] Figure 1B A base product 1B102 (e.g., a handbag), some masks (e.g., product mask 1B104, UV markers 1B106, handle mask 1B108), light map 1B110, and finished product 1B112 are depicted. The depiction of base product 1B102 (handbag) shows a plain cloth handbag. Product mask 1B104 is a binary mask that distinguishes the depiction of the handbag from the background. For example, this mask can be used to mask the depiction of the handbag background and allow customization or enhancement of the depiction of the handbag itself. Or, as Figure 1C As shown, product masks can be used to mask the depiction of the handbag itself and allow for customization or enhancement of the background of the handbag.

[0074] UV marker 1B106 is a marker grid that is used to map the synthesized AI-based image to the front of the handbag. For example, a mask can be used to define an area corresponding to the front of the handbag and paste the AI-based synthesized image onto that area.

[0075] The handle mask 1B108 is a binary mask that allows the depiction of a handbag handle to be distinguished from the background of the handbag handle. For example, the mask can be used to mask out the background of the handbag handle and define the area of ​​the handle itself. In this way, the color of the handbag handle can be customized, etc.

[0076] The lighting map 1B110 can be used to generate light reflection characteristics for the final depiction of the customized product. The map can be used to generate light reflections, shadows, diffuse light, etc. in the final depiction of the customized product.

[0077] The finished product 1B112 is an image in which: (1) the front of the handbag includes an AI-based composite image depicting a colorful picture; (2) the handles of the handbag appear to be green handles, and (3) the entire depiction of the customized product is illuminated using one or more light sources and lighting map 1B110.

[0078] Figure 1C are examples of options and processing by-products. Figure 1B compared to, Figure 1C yes Figure 1B More specifically, new product mask 1B104B is a binary mask that can distinguish the depiction of the handbag from the background of the handbag; however, in mask 1B104B, the handbag itself is masked using black (black pixels) and the background of the handbag is displayed using white (white pixels). According to the masking convention, new product mask 1B104B can be used to mask (e.g., ignore) the depiction of the handbag itself and use the handbag background for applying the new image. Therefore, new mask 1B104B can replace the background of the handbag with the AI-generated synthetic image.

[0079] Using the same masking convention, product mask 1B104 (from Figure 1B ) can be used to mask (e.g., ignore) the background depiction of the handbag and make the depiction of the handbag itself available for application of the new image. Thus, the mask 1B104 can be used to replace the depiction of the handbag itself with the AI-generated composite image.

[0080] In other embodiments, the masking convention may be reversed, so that the masking may have reversed purposes, and so the effects of its application may also be reversed.

[0081] To generate an instant image using the method, a text query may first be generated. The query may include a number of keywords or phrases or similar content. The keywords / phrases may be determined based on the product description of the customized product, information extracted from the user profile, a history of searches performed by the user, etc. This may also include defining the context of the customization process performed on the customized product, and then generating a combination of context data and other keywords / phrases that may be determined from other sources.

[0082] The text query can be provided to a generative AI based application (e.g., SD), which generates output that can include one or more images. SD can be guided based on the input (i.e., the provided text query) to provide a composite image based on images from various databases.

[0083] 2.3. Stable diffusion

[0084] Although the present method does not invent stable diffusion (SD), the present method uses the SD method as a tool to support the inventive aspects of the present method. Therefore, the content disclosed herein includes a brief overview of the SD technology.

[0085] SD is a deep learning text-to-image model that is primarily used to generate detailed images conditioned on textual depictions. The model can also be applied to other applications such as in-painting, out-painting, etc. SD can also be used to generate image-to-image translations based on textual prompts. It was developed by Stability AI in collaboration with a number of academic researchers and non-profit organizations.

[0086] SD is a latent diffusion model used in deep generative networks. The code and model weights used in the SD method have been publicly released and are available via the public domain. Hardware requirements typically include a modest GPU with at least 8GB of VRAM. No access to cloud services is required (this distinguishes SD from other text-to-image models such as DALL-E and Midjourney).

[0087] SD uses the Latent Diffusion Model (LDM) developed by the CompVis group at LMU Munich. In general, a diffusion model is trained to remove Gaussian noise applied continuously on the training images, which can be thought of as a chain of denoising autoencoders.

[0088] Typically, SD consists of three parts: a variational autoencoder (VAE), U-Net, and an optional text encoder. The VAE encoder compresses the image from the pixel space to a latent space of smaller dimensions, thereby capturing more basic semantics of the image. During the forward diffusion process, Gaussian noise is iteratively applied to the compressed latent representation.

[0089] The U-Net block composed of a ResNet backbone denoises the output of the forward-backward diffusion to obtain the latent representation.

[0090] The VAE decoder generates the final image by converting the representation back to pixel space.

[0091] The denoising step can be flexibly conditioned on text strings, images, or other modalities. The encoded conditioning data is exposed to the denoising U-network via a cross-attention mechanism. When conditioned on text, a fixed, pre-trained CLIPViT-L / 14 text encoder is used to convert text cues to an embedding space. Improved computational efficiency for both training and generation is generally an advantage of LDM.

[0092] SD is usually trained on image and caption pairs taken from LAION-5B, a publicly available dataset derived from Common Crawl data scraped from the web. Typically, 5 billion image-text pairs are used, which have been classified based on language and filtered into separate datasets by resolution, predicted likelihood of containing a watermark, and predicted aesthetic scores such as subjective visual quality. The dataset was created by the German nonprofit LAION, which receives funding from Stability AI.

[0093] The stable diffusion model was trained on three subsets of LAION-5B: laion2B-en, laion-high resolution, and laion-aesthetic v25+. A third-party analysis of the model's training data found that in a smaller subset of 12 million images extracted from the broader dataset originally used, about 47% of the image sample size came from 100 different domains, with Pinterest accounting for 8.5% of the subset, followed by WordPress, Blogspot, Flickr, and other websites. The model was trained using 256 Nvidia A100 GPUs on Amazon Web Services, consuming a total of 150,000 GPU hours at a cost of $600,000.

[0094] In some cases, stable diffusion can suffer from some quality degradation and inaccuracies. The initial version of the model was trained on a dataset consisting of 512×512 resolution images, which means that when the user specification deviates from its "expected" 512×512 resolution, the quality of the generated images degrades significantly.

[0095] However, the 2.0 update of the stable diffusion model later introduced the ability to generate images at 768×768 resolution, overcoming some of the above issues.

[0096] The stable diffusion model encountered another challenge in generating human limbs due to the poor quality of limb data in the LAION database. Due to the lack of representative features in the database, the model did not seem to be adequately trained to understand human limbs and faces, and prompting the model to generate such images confused the model.

[0097] The stable diffusion fine-tuned adaptation models created through additional retraining have been used for a variety of different use cases, from medical imaging to algorithmically generated music. However, the fine-tuning process is very sensitive to the quality of the new data; low-resolution images or different resolutions from the original data not only fail to learn the new task, but also reduce the overall performance of the model. Even if the model is additionally trained on high-quality images, it may be difficult for an individual to run the model in a consumer electronics product, such as Nvidia's GeForce30 series, which only has about 12GB, because at least 30GB of VRAM is usually required.

[0098] To address limitations of the initial model training, end users may choose to perform additional training to fine-tune the generated output to match more specific use cases. There are three methods for applying user-accessible fine-tuning to the stable diffusion model checkpoint: (1) Embedding; (2) Hypernetwork; and (3) DreamBooth. This method utilizes a small amount of low-rank adaptation (LoRA) techniques to highlight certain lighting models that fall outside the scope of typical stable diffusion processing when processing very bright and / or very dark images. LoRA techniques can fine-tune based on the prevalence of white in the image and based on the prevalence of black. Used in conjunction with other lighting information, these techniques can make the images produced by the generation pipeline look more refined and vivid. Additional background information on these techniques can be found in, e.g. https: / / huggingface.co / blog / lora , https: / / i.redd.it / well-researched-comparison-of-training- techniques-lora-v0-vl01e5grs6ca1.png? s=cfb3d4eb7d253025ffc68f6791740f773760 4c84 .

[0099] "Embeddings" can be trained from a user-provided collection of images and allow the model to generate visually similar images whenever the embedded name is used in the generation prompt. The embeddings are based on the concept of "textual inversion", where the vector representation of a specific token used by the model's text encoder is linked to a new pseudo-word. Embeddings can be used to reduce bias in the original model or to mimic visual style.

[0100] A "hypernetwork" is a small pre-trained neural network that can be applied at various points in a larger neural network. The hypernetwork steers the results in a specific direction, enabling a stable diffusion-based model to mimic the artistic style of a specific artist, even if the original model cannot identify that artist. They process images by finding key important regions (e.g., hair and eyes), and then inpainting those regions in a secondary latent space.

[0101] DreamBooth is a deep learning generative model that is trained on a set of images depicting a specific topic, and can then be fine-tuned to generate precise, personalized output depicting that topic.

[0102] In the context of the method presented in this paper, the SD method utilizes a text-to-image model to generate detailed images conditioned on textual depictions.

[0103] 3. Select content for the build request

[0104] There are multiple ways to generate a request to the GAI application for providing a composite image. One way includes specifying a text request that includes keywords and / or phrases that specify the subject or genre of the requested composite image. Other ways may include providing a sample image or sample images of the same or similar type to the requested composite image. Other ways may also be used to request a specific composite image.

[0105] The text request that provides the AI-generated synthetic image can be generated using a number of different methods. According to one method, the keywords and phrases of the text query can be generated based on the product definition of the customized product itself.

[0106] By selecting the keywords / phrases of the request (i.e., query) from the product definition content of the customized product. By doing so, the computer collaborative system (e.g., Figure 1A The system 100 in the example does not change the actual definition of the product itself. Instead, the system accesses the entire context, which in this case includes the background and the definition of the customized product itself. Continuing with the handbag example above, this can allow accessing, deleting, or modifying the individual contexts of objects displayed in the customized product depiction, such as accessing, deleting, or modifying the context of the handbag or the person holding the handbag, the background of the handbag depiction, etc.

[0107] The keywords / phrases of the text request can be obtained from the key-value pairs contained in the product definition of the customized product. Key-value pairs are pairs that capture specific characteristics of the customized product, and each customized product may have hundreds or even millions of related key-value pairs. An example of a key-value pair may include a pair of {color, red}, where the term "color" is the characteristic key and the term "red" is the value of the characteristic key. Key-value pairs will be described in detail later.

[0108] The keywords / phrases of the text request can be viewed as a collection of different information that is extracted from the product itself based on key-value pairs or based on annotations attached to the product.

[0109] Keywords / phrases can also be extracted from so-called linked data, including a key-value set that allows the creation of products, associating products with various groups, including social groups, etc. For example, a customized product like a handbag can be associated with various social groups, such as the "beach trip" social group, the "family trip" social group, etc.

[0110] Keywords / phrases of the text request may also be edited by modifying using information from associated events, user groups, event groups, etc. For example, if a user's profile shows that the user (e.g., a teacher) is planning a trip to Santa Cruz Beach, the keywords / phrases of the corresponding text request of the synthetic AI-generated image may also include words such as beach, ocean, sand, etc.

[0111] Figure 1D is an example of customization. In the depicted example, both the front of the handbag and the background of the handbag have been customized using AI-based composite images. Figure 1D The upper part of the 1D102 shows an example of a custom handbag and an example of a custom background related to a beach theme. More specifically, the homepage of the custom handbag displays "Go to the beach instead of teaching", and the background of the custom handbag visualization is related to the beach, sand, ocean, etc.

[0112] According to another example, suppose the user's profile shows that the user is sending Christmas holiday invitations to many of her girlfriends. Then, the keywords / phrases of the corresponding text request of the synthetic AI-generated image may also include words such as Merry Christmas, Christmas decorations, Christmas tree, etc.

[0113] Reference again Figure 1D is an example of customization 1D 104. In the depicted example, both the front of the handbag and the background of the handbag have been customized using AI-based composite images. Figure 1D The lower half of the visualization shows an example of a customized handbag and a customized background related to a Christmas theme. More specifically, the homepage of the customized handbag displays "Merry Christmas", and the background of the customized handbag visualization is related to Christmas, Christmas decorations, a Christmas tree, etc.

[0114] 4. Sample Request

[0115] Keyword / phrase text requests may also be based on attendee information and the physical location of the event. This type of information is typically a rich set of information associated with a product through key values, products, moments, or central uses associated with the product. This may also include key values ​​and structural descriptions around Zazzle "moments" patents, such as U.S. Patent Application Nos. 18 / 120,403; 18 / 120,404; 18 / 120,405; 18 / 120,406; and 18 / 120,407.

[0116] The keywords / phrases of the text request may also be determined based on the user's profile. For example, if the user profile includes information about the user's race and / or gender, the race / gender information may be used as keywords for the text request. Additionally, if the user profile includes information about the user's location, the city the user is in, etc., such information may be used as keywords for the text request.

[0117] Figure 1E is an example of customization. In the depicted example, the text query 1E102 is "First Christmas as a grandma T-shirt", and the corresponding customized product depicts a picture of a grandma wearing a T-shirt whose application displays "First Christmas as a grandma". The background of the depicted model and T-shirt includes a Christmas tree, holiday gifts, etc. All images included in the depiction are AI-based synthetic images obtained separately after submitting the text queries "First Christmas as a grandma" and "T-shirt".

[0118] Figure 1F is an example of customization. In the depicted example, the text query 1E102 is "women's outdoor camping graphic t-shirts", and the corresponding customized product depicts a picture of a woman wearing a t-shirt whose application reads "I'm still playing outdoors". The background of the depicted model and t-shirt includes a forest with trees, moss, dirt roads, etc. All images included in the depiction are AI-based synthetic images obtained separately after submitting the text query "women's outdoor camping graphic t-shirts".

[0119] Figure 1G is an example of customization. In the depicted example, the model’s face, the model’s gender, and the background have been customized using AI-based synthetic imagery. Figure 1G An example 1G102 of custom models and backgrounds used to depict custom T-shirts is shown. Different models include African American women, Caucasian women, Hispanic women, Asian men, etc. Some backgrounds depict urban landscapes - assuming that the user lives in a city. Each custom detail, namely the model's face, the model's body, the background, etc., is an AI-based synthetic image.

[0120] Figure 1H is an example of customization, more specifically, it is Figure 1G Detailed view of the lower left example shown. Figure 1H shows how different AI-based synthetic images can be included in the resulting depiction of a customized product. More specifically, Figure 1H It shows that the model’s face, the model’s gender, and the background have been customized using AI-based synthetic images.

[0121] Figure 1G A custom model and background for depicting a custom T-shirt are shown. More specifically, Figure 1GAn instantly generated synthetic background 1H102 is shown as the background of the model. An instantly generated synthetic image 1H104 is a model for displaying a T-shirt. A real-time generated synthetic image 1H106 is shown as the jeans worn by the model. Of course, the model can also be instantly changed by modifying the text request, such as matching an African American woman, a Caucasian woman, a Hispanic woman, an Asian woman, a man, etc. Similarly, the background can also be instantly changed by modifying the text request, such as matching other cities, rural areas, beaches, etc. The background depicts a city landscape (assuming the user lives in a city). Each customized detail, namely the model's face, the model's body, the background, etc., is an instantly generated AI-based synthetic image.

[0122] Other ways of deriving keywords / phrases for text requests for AI-based synthetic images may also be implemented.

[0123] 5. Base Image

[0124] A base image, such as base product image 1B102, can be obtained by different methods. One method includes taking a photo of the customized product under specific lighting conditions using a camera that typically has high resolution and high contrast.

[0125] Fig. 1I is an example of obtaining a base image. In this example, process 11102 includes preparing a display screen for a customized product (here, a physical handbag), setting up a light source, determining ambient light, and other lighting conditions that are useful and advantageous for depicting the product. In the depicted example, a photographer takes a picture of a handbag displayed on a white background screen. The scene is illuminated by multiple light sources. Light also reflects from the screen panel, etc.

[0126] Another method can also be used to obtain the base image of the customized product. For example, the base image can be a synthetic image, a digital 3D model, etc.

[0127] 6. Use AI to generate synthetic backgrounds / images

[0128] 6.1. Request

[0129] Figure 2A is an example of utilizing an AI-based image generation method. In the depicted example, the composite background and image may be generated as a product of collaboration between the computer collaboration system 100, the AI-based image generator 180, and the user 2E102 operating the user computer 140A.

[0130] In some embodiments, the computer collaboration system 100 communicates with the AI-based image generator 180 using its visualization service 100A.

[0131] The user computer 140A may be used to receive customization requests from the user 2E 102, selections of customized products, and other requests made to the computer collaboration system 100. The user computer 140A may also be used to display graphical visualizations of the customized products, including visualizations of AI-based images generated on the fly that have been merged with or incorporated into visualizations of the customized products.

[0132] The computer collaboration system 100 (or its visualization service 100A) may receive a graphical representation of a customized digital product, such as a pottery product, a linen product, a card, a print, a drawing, etc. The graphical representation of the product may include the product's coded information and a product description - including key-value pairs described later.

[0133] In some embodiments, the graphical representation of the customized digital product includes product description data of the customized digital product. The product description data includes multiple parameters of the customized digital product, each of the multiple parameters of the customized product is a key-value pair. A key-value pair is a data structure that includes an attribute key and a corresponding attribute value.

[0134] The visualization service 100A determines to customize one or more regions in the product description based on the received product description and user profile, events, etc. These regions will be customized when the graphical representation of the customized digital product is displayed on the display device.

[0135] The regions may be determined based on information contained in the product graphical representation, key-value pairs, user profiles, etc. The regions may include a background displayed behind the product description, or other regions displayed behind or within the product description (or descriptions). For example, if the product is a Christmas card, then the region may be a background region displayed behind the custom Christmas card depiction. According to another example, if the product is an invitation card, then one region may be a region displayed above the invitation card depiction, while another region may be a region displayed on the invitation card itself.

[0136] For each region, the visualization service 100A determines a keyword set / phrase set specific to the customized digital product and the customized digital product. The keyword set / phrase set may include one or more key-value pairs extracted from the product description data. The phrase set may also include one or more information extracted from the user profile of the user who customized the customized digital product. The phrase set may further include one or more information extracted from the event library related to the customized digital product. Various methods for obtaining keywords / phrases have been introduced above.

[0137] For each region, a request 190 is created using the corresponding keyword / phrase.

[0138] The request can be a text request and can be related to a specific region for which the request is created. This request is also called a composite image generation request. Multiple requests can be generated for a region.

[0139] All requests are sent by the visualization service 100A to the AI-based image generator.

[0140] AI-based image generator

[0141] The AI-based image generator 180 is configured to receive a composite (or synthesized) image request, generate a composite / synthesized image in response to the received request, and provide the generated composite / synthesized image to the visualization service 200A.

[0142] Figure 2B is an example of merging AI input and corresponding AI output with a customized image. For a given customized image, the AI-based image generator 180 may receive multiple requests 190A, 190B, ... 190N. These requests may be related to a single area in the customized image; however, it is also possible to receive multiple requests for a single area in the customized image.

[0143] The AI-based image generator 180 may be configured to access one or more databases, such as database 174A, database 17, etc. The database may store SD-related files, repository images, training data, training models, learning models, etc.

[0144] In response to receiving the requests 190A- 190N, the AI-based image generator 180 may generate one or more instantly generated composite images 190A- 190N.

[0145] 6.3. Synthetic Images

[0146] The composite image may be generated based on the keywords / phrases included in the corresponding request. For each request, one or more composite images may be generated. As previously described, each instant composite image is generated / synthesized from one or more images retrieved and synthesized from the database(s) 174. Therefore, the composite image 190 is unlike any image stored in the repository.

[0147] For each request 190, upon receiving the request, the AI-based image generator 180 generates a customized composite image based at least in part on the request and the image repository (as described above). The customized composite image is not a copy of any image in the image repository. Instead, the customized composite image is generated by synthesizing a set of images selected by the AI-based image generator from the image repository and synthesizing the selected images using the SD method described above.

[0148] Custom composite images can be generated by compositing two or more images extracted from an image library that are annotated with phrases that match one or more phrases in a phrase set. Images in the image resource library can be annotated with corresponding annotations. Custom composite images can be generated simultaneously by updating a custom digital product display on a display device. Therefore, they are called composites or composite images generated on the fly.

[0149] Refer to Figure 2A Once the AI-based image generator 180 generates the customized composite image, the generator transmits the image to the computer collaboration system 100, and the virtualization service stores the image as an instantly generated composite image 192.

[0150] Upon receiving the composite image 192 generated on the fly, the visualization service 100A matches the received image with the corresponding region, contains the image in the corresponding region, and generates a graphical visualization of the customized product and the corresponding composite image.

[0151] The visualization service 100A then displays the graphical visualization of the customized digital product and the customized composite image in the corresponding area on the display device.

[0152] The graphical visualization and the graphical representation of the customized digital product may be transmitted or otherwise provided to a manufacturing product rendering unit of the manufacturing entity. The manufacturing product rendering unit may render the graphical visualization and the representation of the customized digital product, for example, printing the graphical representation of the customized digital product using printing instructions of the manufacturing product rendering unit to generate a physical representation of the customized digital product.

[0153] 7. Example Manufacturing System

[0154] In some embodiments, a method is implemented in a product collaboration platform to use products as keys for context-based high-quality rendered image generation and manufacturing output for generating customized products. The platform allows users, designers, customers, support engineers, etc. to design and create digital designs of products. An example of a computing environment for creating digital designs, manufacturing products, etc. will be described in Figure 3A Described in.

[0155] 7.1. Description of the structure of the authoring tool

[0156] In some embodiments, the product visualization platform is configured to execute one or more software applications to generate high-quality renderings and manufacturing outputs of customized products. Generating high-quality renderings and outputs may include performing multiple visualization tasks and production tasks. These tasks may be completed by executing software applications as a series of sequential and / or parallel processes. As described below, these applications are also referred to as authoring applications or authoring tools.

[0157] The authoring tool can receive as input an initial description of the product, which will be displayed by the platform and then made available for customization. Based on the initial description, the tool can generate layout or manufacturing files, which include a high-level representation of how the product should be manufactured or aesthetically pleasing and what customizable parts the product has. Layout and manufacturing files are described later.

[0158] The manufacturing file may also include a mark, etc., and the file with the mark may be sent to the manufacturer. The mark may be a two-color checkerboard pattern consisting of a plurality of squares, wherein each square is a single color and each adjacent square is an opposite color. For example, the mark may be a two-color checkerboard pattern consisting of solid color squares, wherein each square is a single color and each adjacent square is an opposite color. The marked portion of the input image may be divided into a set of regions, wherein each region in the set of regions represents a corresponding square of the mark. In other embodiments, the mark may include different patterns, shapes, or (one or more) colors. These marks have been disclosed in, for example, U.S. Patent Application Nos. 9,852,533B2 and 10,283,165B2.

[0159] After sending the product manufacturing file to the manufacturer, the manufacturer can generate the physical product including the marking and send the physical product to the platform operator.

[0160] Upon receiving the physical product with the mark, the platform operator can use a device such as a digital camera to take a set of high-quality photos that depict the physical product and the corresponding mark. These photos are then provided as input to the authoring tools executed on the platform. For example, photos containing the mark can be used as input to generate various views of the product and depict the product with different surface treatments, different patterns, different colors, different shades, etc.

[0161] In some embodiments, the visualization platform uses various views (e.g., configurable images, layouts, and design views (described later)) to depict the product and allow for the generation of outputs that can then be displayed, for example, on a website generated by the visualization platform and offered to customers for personalization, customization, and potential purchase. The different views are described below.

[0162] Various depth maps and laser-based images may also be used. Devices that capture information about the physical appearance or structure of a product may include cameras, scanning devices (e.g., laser scanners, iPhones (time-of-flight imaging), Kinects (structured infrared light patterns), etc. Depth maps (generated using, for example, Microsoft's Kinect hardware and laser scanners) can be used as scanning devices in testing and asset production. In the context of this disclosure, they operate similarly to cameras. They can be configured to capture physical attributes or surface features of a product. Depth maps and laser scans can then be used for asset generation in authoring tools.

[0163] If the customer completes the customization of the product, such as placing an order, then the data describing the customized product can be transmitted to, for example, a print server. The print server can use this data to generate a manufacturing output file. The output file can contain manufacturing instructions that the manufacturer can use to manufacture the customized product ordered by the customer. The manufacturing instructions will be described later.

[0164] 7.2. Design assets

[0165] In some embodiments, the product visualization platform is configured to execute a software application designed to generate high-quality renderings and manufacturing outputs of customized products. For example, high-quality renderings can be used to generate descriptions of various products and product components. These products are also referred to herein as assets. Assets can be displayed on a website or other service, where users can interact with digital files featuring assets to facilitate customer personalization, customization, and ordering of products. For example, customization of a product may involve using a high-quality rendering so that the product can be displayed using the functionality of a website or selected display service.

[0166] However, before high-quality renderings of an asset can be produced, the asset needs to be designed and then tested to determine if a manufacturer can build a physical product that corresponds to the asset according to the designer's specifications.

[0167] 7.3. Asset definition data

[0168] Asset definition data is data that parameterizes descriptions of assets and / or customized products. In some embodiments, asset definition data includes product description keys, such as those disclosed in U.S. Patent Application No. 17 / 458,116, U.S. Patent Application No. 17 / 501,513, U.S. Patent Application No. 17 / 387,154, U.S. Patent Application No. 17 / 384,636, U.S. Patent Application No. 17 / 501,451, U.S. Patent Application No. 16 / 867,898, and U.S. Patent Application No. 11,080,912.

[0169] In addition to product description key values, manufacturing constraints can also be used. Manufacturing constraints can be used to specify the scope and boundaries of the equipment used by manufacturers involved in producing the product. Manufacturing instructions are usually compatible with the configuration of, for example, manufacturing servers (e.g., print servers), which process the manufacturing instructions and generate output that meets the requirements set by the customer.

[0170] Manufacturing instructions typically capture all the information that needs to be provided to a manufacturer so that the manufacturer can produce / manufacture the product customized by the user and / or designer. Manufacturing instructions capture key-value pairs (described in detail later) in a form that is understandable by the equipment deployed by the manufacturer. As described later, two or more customized products may each have a single set of key-value pairs; however, not all key-value pairs in the corresponding single sets of key-value pairs are the same, only some of the key-value pairs are the same. For example, a series of coffee mugs printed with a specific floral pattern may share or have a similar subset of key-value pairs to capture the details of the specific floral pattern.

[0171] Certain details of a group of specific customized products may be captured by a subset of key-value pairs that may be unique to each specific customized product. This subset of key-value pairs may be contained in a so-called dictionary and marked as reusable key-value pairs. The dictionary may be stored, for example, in a storage or memory unit of the product customization platform, which will be described later.

[0172] Likewise, one or more specific manufacturing constraints in the plurality of manufacturing constraints for manufacturing a group of specific customized products may be specific to each specific customized product. One or more specific manufacturing constraints may be included in a dictionary and marked as reusable manufacturing constraints.

[0173] In reality, when talking to a manufacturer, it may be difficult to describe the necessary manufacturing constraints in a way that the products made by the manufacturer match the customized products (described using key-value pairs). Therefore, it is necessary to use a dictionary with specific manufacturing instruction tags (corresponding to a specific subset of key-value pairs), and then select a short list of manufacturing instructions from the specific manufacturing instruction tags and use it as a starting point to capture the characteristics of the customized product. The manufacturing instructions in the short list can then be refined and further fine-tuned until a precisely manufactured product can be produced.

[0174] 7.4. User Equipment

[0175] Figure 3A is a block diagram illustrating an example environment for designing and manufacturing products. Figure 3A Several examples of users 10 are described. Each user 10 can use their own or shared computer equipment. Examples of user equipment will be described in Figure 3B Described in (see devices 140A-G).

[0176] In some embodiments, examples of user 10 are determined based on roles that may be assigned to the user. Role examples 10A may include shopper, customer, designer, client peer, user support engineer, receiver, and the like.

[0177] 7.5. Client Application

[0178] Figure 3A The client 12 in the reference numeral 10 refers to a client application implemented in the client server 14 and configured to support requests received from the user 10A. Non-limiting examples of the client 12 may include an iOS application 12A, an Android application 12B, a Web application 12C, and the like.

[0179] 7.6. Front-end Server

[0180] Front-end server 14 refers to a computer-based server that is configured to process requests received from client 12 and, in many cases, interact with core service 16 to further resolve these requests. Examples of front-end server 14 include one or more WWW servers 14A, one or more application servers 14B, and one or more encryption servers 14C. Encryption server 14C may be configured to provide services for encrypting / decrypting, transmitting, or otherwise transmitting data between client 12 and core service 16. Figure 3A A service for encryption of data transmitted between entities depicted in .

[0181] 7.7. Core Services

[0182] Figure 3A The core services 16 in refer to servers and services implemented in a role-based collaboration platform that is configured to provide functionality for creating and designing digital designs, processing collaboration requests, and facilitating customization requests received from users 10.

[0183] In some embodiments, the customization process performed by one of the users 10 and aimed at generating a digital design of a customized product is captured in so-called product description data, which can then be translated into a manufacturing description including the product and manufacturing instructions.

[0184] The product and manufacturing instructions may include digital design specifications, data and codes required to manufacture the customized product. This may include instructions for generating, for example, 3D geometry of a digital final product. This may also include instructions for generating 2D and / or 3D patterns that can be used to cut, cast or form physical parts of a physical final product. The patterns may be parametric, that is, they may have parameters that adjust the form of the pattern through encoded relationships for specific needs.

[0185] For example, a 2D pattern set for a T-shirt based on a size grading can be turned into a parameterized pattern by interpolating the graded curvature. A single parameterization value can set this automatic grading. The single parameterization value is usually called 'size'.

[0186] Product instructions may also include models, including 2D models and 3D models for forming parts of the product by additive manufacturing or subtractive manufacturing. The models may be parameterized, that is, they may have parameters that adjust the form of the model for specific needs through encoded relationships. For example, a set of 3D models may represent bicycle helmets. Each model may conform to a statistically normative human head of a specific age. The encoded relationships between the models may allow the set of models for a specific age to be interpolated. A single parameterized value may set automatic interpolation. The single parameterized value in this case is typically called 'age'.

[0187] The product instructions may also include material properties, such as the physical material used to form the product from the pattern. Some material properties may be parameterized, ie, they may be selected or changed during manufacturing time.

[0188] The properties may also include body color. For example, the color of the fabric may be selected to make a T-shirt. According to another example, the color of the plastic may be selected to make a bicycle helmet.

[0189] The properties may also include body texture, for example, the fabric weave of a t-shirt may be specified as smooth or rigid. For example, the surface of a plastic bicycle helmet may be polished or satin. Each property is necessarily specific to each type of material. Examples of materials and properties may include fabric (e.g., woven or knit type), fiber type (cotton, wool, linen, polyester, polypropylene), yarn size, yarn count, color, overall design (ikat, knit, tapestry), bolt width, selvedge type, finish (hand), etc.

[0190] The properties may also include plastic, which may include sub-properties such as color, surface quality (bidirectional brightness function), melting point, impact resistance, molding method (thermoplastic, casting), type (acrylic, polypropylene, etc.), etc.

[0191] The properties may also include metals, which may include sub-properties such as type (aluminum, steel, copper, brass, etc.), color, surface quality (e.g., bidirectional brightness function), melting point, tensile strength, shear strength, toxicity, etc.

[0192] The properties may also include nonwovens specified by type (paper, felt, Tyvek, etc.), color, surface quality (e.g., bidirectional brightness function), surface type (hot pressed, natural, textured, etc.), weight per square meter, acid content, compatible media, coatings, etc.

[0193] The properties may also include metamaterials, which may be described as a combination of multiple materials created during a manufacturing process. For example, during fused deposition manufacturing, plastics with multiple properties may be mixed to provide a physical product with graded flexibility, durometers, etc. According to another example, during laser sintering of metals, a mixture of metal alloys with multiple properties may be deposited, resulting in a product composed of a graded metal property. According to yet another example, during high-resolution UV printing, layers of UV-cured materials with different refractive indices may be deposited, resulting in multiple optical effects.

[0194] The properties may also include decoration, such as trim color, design, and decorations applied. Trim color may indicate the color of the edge around the sleeve of a T-shirt, the color of the trim around the base of a bicycle helmet, etc.

[0195] The design may indicate a custom printed front of a t-shirt, a custom thermal printed design on the side of a bicycle helmet, etc.

[0196] The decorations applied can be rhinestones on a T-shirt, holographic sequins on a bicycle helmet, etc.

[0197] Some properties may apply to a large class of products and may allow a limited set of properties to be transferred from one product to another. Other properties may be specific to a particular product or manufacturing method.

[0198] It will be appreciated that much of the novelty of the system and method lies in enumerating the constraints imposed by manufacturing a particular customized product and crafting these constraints into a set of product option key-value pairs. The manufacturing constraints are propagated throughout the system and method, and by using these product option keys, a range of customized products that satisfy these physical constraints are allowed to be manufactured.

[0199] Reference again Figure 3A In some embodiments, core services 16 refer to services implemented in a role-based collaboration platform. In the depicted example, core services 16 may be provided by one or more real view (RLV) servers 16A and a product option framework 16AA. Both RLV servers 16A and product option framework 16AA may use one or more data layer databases 16B, including RLV data 16C, product option database 16D, transaction database 16E, etc.

[0200] In some embodiments, core services 16 may also utilize internal tools 16F, such as "Phlow" computational photography tools 16E, customer support tools 16G, launch pad tools 16H, and the like.

[0201] The product options framework 16AA is also referred to as a persistent design data framework. The framework data may include a product option set, which may include a product option set related to a specific product type. It typically contains product instructions for manufacturing or producing a product (e.g., collaboration component 106 in FIG. 2 ).

[0202] In some embodiments, the product options framework 16AA is configured to provide services for converting ProductOption key-value pairs (i.e., manufacturing constraints) from one product to another. For example, converting ProductOption key-value pairs from one product to another may require converting color spaces (i.e., converting sRGB to CMYK US Web Coated (SWOP) v2), converting images from raster to vector, and / or resizing images to fit.

[0203] Figure 3B Describes a use case for the Product Options Framework in detail.

[0204] In some embodiments, there are two basic types of product instructions: (1) fixed (including instructions for fixed and non-customized products), and (2) customized (including logic to convert user interface parameters into product instructions).

[0205] In some embodiments, a product option set may contain logic that can enumerate each customizable option to provide a complete user interface for changing parameterized product instructions.

[0206] The instructions for making a customized product are usually parameterized. The parameters include the size of the customized product (which can be multi-dimensional and include width, height, depth). The parameters can also be related to human body size or age. The parameters can also be customized and based on biometric information.

[0207] The parameters may also include component body color, component body texture, trim body color, trim body texture, design area, etc.

[0208] In some embodiments, product options can be represented as key-value pairs. Key-value pairs are tags that can span individual products and represent a class of products. The keys of the pair can include material type, color, size, etc.

[0209] The value in the key-value pair is a specific discrete or continuous value that sets the manufacturing instructions. Examples of discrete (enumerated) values ​​may include discrete types of fabrics, such as cotton, cotton-polyester blends, silk, etc. Discrete values ​​may also include specific colors, such as white, navy, black, etc.

[0210] Examples of continuous values ​​for a key-value pair may include a single element (e.g., a length or a ribbon), a vector (e.g., the size of a print box (width in inches) or height in inches), or the sizes of boxes for European countries, such as the sizes of boxes for the European Union (EU) (width in millimeters, height in millimeters, depth in millimeters).

[0211] The values ​​may also reference known file types, such as an image for a design on a T-shirt, an embroidery file for the back of a jacket, an engraved design for a bracelet, etc.

[0212] In some embodiments, the value in the key-value pair may include a set of graphical primitives for the design, such as images, lines, circles, rectangles, text, groups, etc.

[0213] Product option key values ​​can have default values. Default values ​​are preset values ​​that produce a product without changing any key value pairs through customization. When key values ​​change, they may generate a product option framework event chain. The product option framework event chain is a time-ordered log of each key value change.

[0214] The product type itself can be represented by a product option key. Using this option type, one product type can be related to another product type through a well-known relationship.

[0215] In some embodiments, a product option framework event chain includes one or more products, and the chain may represent or remember an event. The product may represent or remember an event. Examples of events may include weddings, birthdays, anniversaries, graduations, national holidays, reunions, etc.

[0216] Many products are suitable for event chains. For example, products suitable for wedding events may include the following: save the date products, invitations, RSVP, bachelor party invitation products, bachelorette party invitation products, party favor products, gift registry cards, place cards, napkins, wedding programs, wedding favors, thank you cards, wedding keepsake photos, etc.

[0217] Examples of products that may be suitable for birthday events may include the following: invitations, RSVPs, party bags, party keepsake photos, etc.

[0218] In some embodiments, in a product option set event chain, a key-value pair encodes the next product in the chain. For example, an invitation can be chained to an RSVP card. The key-value can also encode the role of the chained event. For example, the chained RSVP card key-value can further include the recipient of the invitation as the sender role of the RSVP card.

[0219] Key-value pairs can also encode shared properties that are used to set chained product properties. For example, the design of an invitation can be shared with an RSVP card. Key-values ​​can also encode the timing of chained products. Typically, event chain properties are customized (parameterized) and they can be changed by the product designer to suit a specific set of products.

[0220] In some embodiments, the product option framework is configured to generate a product option framework user interface. Accordingly, each product option set is associated with logic and code to build user interface elements for each parameterized product option. In addition, each product option set includes style hints so that each user interface element can be placed smartly to produce a high-quality user experience.

[0221] Typically, UI elements are designed to match each category of values ​​found in all products covered by the Product Options Framework. As product categories expand, new UI elements may be added. UI elements may include design views, color editors, font editors, size selectors, texture selectors, text editors, fabric swatch selectors, product realistic views, and more.

[0222] In some embodiments, the product options framework cooperates with a user product renderer that can be implemented, for example, in the real view server 16A. The user product renderer can be configured to render a view of a customized product as if it had been manufactured. Typically, it uses a keyed set of product options as input. It creates one or more runtime assets using computational photography of manufactured products.

[0223] 7.8. Example Manufacturing System

[0224] Reference again Figure 2A , manufacturing instructions may be transmitted from core services 16 to manufacturing 18, which may include one or more manufacturing servers 16AAA. Servers 16AAA may receive manufacturing instructions, process the instructions, and communicate with supplier manufacturing applications 18A to generate, for example, manufacturing data, operator directions, tool / machine code, etc. The applications may also generate information about order status, packing lists, production orders, etc. Some of the information may be communicated to a carrier (transportation) service selected to deliver the final product to the recipient.

[0225] The final product can be manufactured using markings. For example, the body color of a product can be marked by specifying a different and separate color for the BodyColor key. The trim color of a product can be marked by specifying a different and separate color for, for example, the TrimColor key pair. The design area of ​​a product can be marked by specifying a specific marker type (e.g., a removable linear marker, a numeric marker, or a numeric field marker) for the design area image.

[0226] The construction of so-called product reality geometry can be performed using computational photography techniques. For example, the BodyColor region of a surface can be identified and layered into a custom computer graphics shader for rendering light, color, and texture. The surface of each TrimColor region can be identified and layered into a custom computer graphics shader for rendering light, color, and texture. The rendering of the product reality can be performed by setting its product option key and shading each layer it constructs.

[0227] 7.9. Example Manufacturing Process

[0228] As described above, the manufacturing process may involve manufacturing a digital product as well as manufacturing a physical product. Since the manufacturing instructions for generating a product are generated based on a plurality of key-value pairs for the digital design of the product, in some cases, the same manufacturing instructions may be used to manufacture a digital product as well as manufacture a physical product.

[0229] In some embodiments, the product options framework (in Figure 3B Detailed description in

[0066] ) builds an interface for a key called OutputStyle. The interface for the OutputStyle key can allow the designer (or any other collaborator) to select a value for the medium used to render the interactive design. These choices can include JPEG_Image, GIFF_Image, and H264_Video.

[0230] If the designer selects the GIFF_Image option, the Product Options Framework can send instructions to the Manufacturing System to iterate through each key value in the KeyValueJournal and, for each key, render the state of the customized physical product with the modification as an image in sRGB 32-bit RGBA format using the User ProductRenderer. The Manufacturing System can then store the rendering in a local image cache.

[0231] The manufacturing system may then iterate through the images stored in the local image cache and determine an optimal color palette for the set of images.

[0232] The manufacturing system may then convert the image in the local image cache from 32-bit RGBA format to 8-bit indexed color.

[0233] The manufacturing system can then embed a digital watermark encoding the UUID of the input KeyValueJournal in an 8-bit indexed color image cache.

[0234] Next, the manufacturing system can begin encoding the image file. For example, the manufacturing system can write the header bytes; write the logical screen descriptor bytes; write the found palette as the gif global color table; write the gif 8-bit character application name; and embed metadata as a comment (or watermark) encoding the UUID of the input KeyValueJournal.

[0235] Next, the manufacturing system sets FrameCount to 1 and continues to process each frame in the image file. The process includes checking if there is an image in the 8-bit indexed color image cache; and if so, the manufacturing system continues; otherwise, the manufacturing system proceeds to acquire the next image.

[0236] Next, the manufacturing system writes the gif graphics control description of FrameCount and then caches the first 8-bit indexed color image into blocks of 255 LZW compressed bytes.

[0237] Next, the manufacturing system writes the compressed bytes and removes the first image from the 8-bit indexed color image cache. The manufacturing system then increments the frame count and repeats the process for the next frame.

[0238] Once the manufacturing system has processed all frames, the manufacturing system writes a file terminator (eg, an ASCII code for zero) to the image file and outputs the manufactured GIF product.

[0239] At this point, execution of the manufacturing instructions for the purpose of manufacturing the product ends, and manufacturing of the product is completed.

[0240] 8. Sample Product Collaboration Platform

[0241] In some embodiments, the methods presented herein are implemented in one or more computer systems that host a product collaboration platform. Alternatively, the methods may be implemented in one or more computer systems that communicate with the collaboration platform but do not actually host the platform itself. For clarity of description, it is assumed that the computer environment supporting the methods presented herein is implemented in a product collaboration platform.

[0242] Figure 3B is a block diagram showing an example of a role-based collaboration platform 1. Figure 3B In the example shown, the computer collaboration system 100 includes a user profile database 102, a global key-value database 103, a product data definition database 104, a property tree database 105, a collaboration component 106, a product option framework 110, a property engine 108, one or more processors 120, one or more memory units 122, and one or more frameworks 129 to 136.

[0243] In addition, the computer collaboration system 100 includes a collaboration server 155 (including a monitoring system 156 and a request analyzer 159), one or more blacklist databases 163, one or more action databases 162, a request generator 152, a communication processor 153, and an image requester 158 and an image superimposer 160. The computer collaboration system 100 can communicate with one or more user computers 140A-140G directly or via one or more communication networks 130, all of which will be described in detail later.

[0244] This article provides Figure 3B The computer collaboration system 100 is shown to illustrate a clear example and should not be considered limiting in any way. Other computer collaboration systems may include additional components, such as computer servers, various input / output devices, and Figure 3B Additional software and / or hardware components not depicted in the drawings. Other computer collaboration systems may be implemented in cloud-based systems, distributed computer networks, etc. Other computer collaboration systems may include Figure 3B The exemplary computer collaboration system 100 is shown with fewer components.

[0245] Figure 3B Some of the components described in are also described in US Patent Application No. 18,120,402. Components such as request generator 152, image requester 158, and image superimposer 160 have been described above.

[0246] 9. Example Flowchart

[0247] Figure 4 is a flow chart depicting an example process of using a product as a key for context-based image generation. The method can be performed by the computer collaboration system 100 or its visualization service 100A. Both the computer collaboration system and the visualization service are configured to communicate with the AI-based image generator 180 and request and receive AI-based composite images. To simplify the description, it is assumed that the following steps are performed by the visualization service.

[0248] In step 400, a visualization service of a computer collaboration system receives a graphical representation of a customized digital product. The graphical representation of the customized digital product may be constructed using a series of rendering and compositing actions derived from product description data of the customized digital product.

[0249] The composition action includes a manufacturing instruction. The product description data includes a plurality of parameters of the customized digital product. Each of the plurality of parameters of the customized product is a key-value pair. A key-value pair is a data structure including an attribute key and a corresponding attribute value.

[0250] In step 402 , a visualization service determines one or more regions for customizing a region when displaying a graphical representation of a customized digital product on a display device.

[0251] In step 404 , the visualization service determines, for each of the one or more regions, a set of phrases specific to the customized digital product and a customization process for customizing the customized digital product.

[0252] The phrase set includes one or more key-value pairs extracted from the product description data. The phrase set may also include one or more pieces of information extracted from a user profile of a user who customized the customized digital product. The phrase set may also include one or more pieces of information extracted from an event repository associated with the customized digital product.

[0253] For each region and based on the corresponding phrase set, the visualization service generates a composite image generation request in step 406. Also in this step, the visualization service transmits the composite image generation request to an artificial intelligence (AI) based image generator configured to generate a composite image.

[0254] The request is transmitted to the AI-based image generator so that the AI ​​image generator generates, for each region and each request for that region, a customized composite image based at least in part on the composite image generation request and the image repository. All customized composite images are not copies of any image in the image repository. The AI ​​image generator then transmits the customized composite image to the computer collaboration system.

[0255] The customized composite image is generated by composite two or more images extracted from an image library, the images being annotated with phrases matching one or more phrases in a phrase set. The images in the image library are annotated with corresponding annotations. The customized composite image is simultaneously generated by updating a display of the customized digital product on a display device.

[0256] In step 408, the visualization service checks whether the AI ​​image generator has received any synthesized images in response to the sent request. If an image has been received, the visualization service proceeds to step 410. Otherwise, the visualization service continues to check in step 406.

[0257] In step 410 , the visualization service pairs the received customized composite image with its corresponding region and includes the customized composite image in the corresponding region.

[0258] Also in this step, the visualization service displays the graphical visualization of the customized digital product on the display device, along with the customized composite image contained in the corresponding region.

[0259] Additionally, if certain expected custom composite images have not been received, the visualization service may wait for those images in step 408, regenerate the request (or requests), or end the process.

[0260] 10. Implementation Examples

[0261] Although the flowcharts of the present application depict a particular set of steps in a particular order, other implementations may use fewer or more steps in the same or different order than those depicted in the figures.

[0262] According to one embodiment, the technology described herein is implemented by one or more special-purpose computing devices. The special-purpose computing device may be hard-wired to perform the technology, or may include a digital electronic device (e.g., one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are permanently programmed to perform the technology), or may include one or more general-purpose hardware processors that are programmed to perform the technology according to program instructions in firmware, memory, other storage devices, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to complete the technology. The special-purpose computing device may be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that incorporates hard-wiring and / or program logic to implement the technology.

[0263] Figure 5 5 is a block diagram depicting an exemplary computer system 500 upon which an embodiment may be implemented. The computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a processor 504 coupled to the bus 502 for processing information. The computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 502 to store information and instructions to be executed by the processor 504. The main memory 506 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 504. The computer system 500 also includes a read-only memory (ROM) 508 or other static storage device coupled to the bus 502 for storing static information and instructions for the processor 504. A storage device 510, such as a magnetic disk or optical disk, is provided and coupled to the bus 502 for storing information and instructions.

[0264] Computer system 500 may be coupled to a display 512 (e.g., a cathode ray tube (CRT)) via bus 502 for displaying information to a computer user. Although bus 502 is shown as a single bus, bus 502 may include one or more buses. For example, bus 502 may include, but is not limited to: a control bus through which processor 504 controls other devices within computer system 500; an address bus through which processor 504 specifies memory locations for instructions to execute; or any other type of bus for transmitting data or signals between components of computer system 500.

[0265] An input device 514, including alphanumeric and other keys, is coupled to the bus 502 for communicating information and command selections to the processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, which is used to communicate direction information and command selections to the processor 504 and to control cursor movement on the display 512. Such input devices typically have two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane.

[0266] The computer system 500 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware and / or program logic, or computer software that is combined with the computer system to make or program the computer system 500 into a special purpose machine. According to one embodiment, these techniques are performed by the computer system 500 in response to the processor 504 executing one or more sequences of one or more instructions contained in the main memory 506. Such instructions may be read into the main memory 506 from another computer-readable medium (e.g., storage device 510). The execution of the sequence of instructions included in the main memory 506 causes the processor 504 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used instead of software instructions or in combination with software instructions to implement the embodiments. Therefore, the embodiments are not limited to any specific combination of hardware circuitry and software.

[0267] As used herein, the term "computer-readable medium" refers to any medium that participates in providing data that causes a computer to operate in a particular manner. In an embodiment implemented using computer system 500, various computer-readable media, for example, participate in providing instructions to processor 504 for execution. Such media can take a variety of forms, including but not limited to non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of computer-readable media include, for example, floppy disks, floppy disks, hard disks, tapes or any other magnetic media, CD-ROMs, any other optical media, RAMs, PROMs, and EPROMs, FLASH-EPROMs, any other memory chips, or storage boxes, or any other media that a computer can read.

[0268] Various forms of computer readable media may be involved in transferring one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be loaded on a disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A local modem of the computer system 500 may receive the data on the telephone line and use an infrared transmitter to convert the data into an infrared signal. An infrared detector may receive the data in the infrared signal and appropriate circuitry may place the data on the bus 502. The bus 502 transfers the data to the main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by the main memory 506 may be optionally stored on the storage device 510 before or after execution by the processor 504.

[0269] The computer system 500 also includes a communication interface 518 coupled to the bus 502. The communication interface 518 provides a two-way data communication coupled to a network link 520, which is connected to a local network 522. For example, the communication interface 518 can be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a telephone line of a corresponding type. As another example, the communication interface 518 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN. A wireless link can also be implemented. In any such implementation, the communication interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0270] Network link 520 typically provides data communications through one or more networks to other data devices. For example, network link 520 may provide a connection through local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. ISP 526 in turn provides data communication services through the global packet data communications network now commonly referred to as the "Internet" 528. Both local network 522 and Internet 528 use electrical, electromagnetic or optical signals that carry digital data streams.

[0271] Computer system 500 may send messages and receive data, including program code, through the network(s), network link 520, and communication interface 518. In the Internet example, server 530 may transmit code requested by an application program through Internet 528, ISP 526, local network 522, and communication interface 518. The received code may be executed by processor 504 as it is received, and / or stored in storage device 510 or other non-volatile storage for later execution.

[0272] In the foregoing description, embodiments have been described with reference to many specific details, which may vary depending on the implementation. Therefore, the only and exclusive indicator is the set of claims proposed from this application, which set of claims is in the specific form in which such claims are proposed, including any subsequent corrections. Therefore, limitations, elements, properties, features, advantages or attributes not expressly recited in the claims should not limit the scope of such claims in any way. Therefore, the description and drawings should be regarded as illustrative and not restrictive.

Claims

1. A method of using a customized product as a key for context-based image generation, the method comprising: receiving a graphical representation of a customized digital product through the visualization services of a computer collaboration system; determining, by the visualization service, one or more regions for customizing regions when displaying a graphical visualization of the customized digital product on a display device; For one of the one or more regions: determining, by the visualization service, a set of phrases specific to the customized digital product and a customization process for customizing the customized digital product; generating a synthetic image generation request based on the phrase set; transmitting the composite image generation request to an artificial intelligence (AI) based image generator configured to generate a composite image; Enable the AI-based generator: generating a custom composite image based at least in part on the composite image generation request and an image repository; wherein the custom composite image is not a copy of any image from the image repository; transmitting the customized composite image to the computer collaboration system; including the customized composite image in the region; The customized composite image in the area and the graphical visualization of the customized digital product are caused to be displayed on the display device.

2. The method according to claim 1, wherein: said graphical representation of said customized digital product is constructed using a series of rendering and compositing actions derived from product description data of the customized digital product; Wherein, the synthesis action includes manufacturing instructions; Wherein, the product description data includes a plurality of parameters of the customized digital product; Wherein, each of the plurality of parameters of the customized product is a key-value pair; The key-value pair is a data structure including an attribute key and a corresponding attribute value.

3. The method according to claim 2, wherein: The phrase set includes one or more key-value pairs extracted from the product description data.

4. The method according to claim 2, wherein: The phrase set includes one or more items of information extracted from a user profile of a user who customized the customized digital product.

5. The method according to claim 2, wherein: The phrase set includes one or more pieces of information extracted from an event repository associated with the customized digital product.

6. The method according to claim 2, wherein: The customized composite image is generated by composite two or more images extracted from the image repository, the two or more images being annotated with phrases matching one or more phrases in the phrase set; Wherein, the pictures in the picture library are annotated using corresponding annotation pairs; Wherein, the customized composite image is simultaneously generated by updating a display of the customized digital product on the display device.

7. The method according to claim 1, further comprising: transmitting said graphical representation of said customized digital product to a manufacturing product rendering unit; To enable the manufacturing product rendering unit: rendering said graphical representation of said customized digital product; The graphical representation of the customized digital product is printed using one or more printing instructions of the manufactured product rendering unit to generate a physical representation of the customized digital product.

8. One or more non-transitory computer-readable media storing one or more instructions that, when executed by one or more processors, cause the one or more processors to perform steps 1-7 of claim 1.

9. A customized product computer system generator, comprising: Memory unit; one or more processors; as well as A customized product computer stores one or more instructions, which, when executed by one or more processors, cause the one or more processors to perform steps 1-7 of claim 1.

Images