Visual Asset Development Using Generative Adversarial Networks

Abstract

A virtual camera captures first images of three-dimensional (3D) digital representations of a visual asset from different viewpoints and under different lighting conditions. The first images are training images stored in a memory. One or more processors implement a generative adversarial network (GAN) that includes a generator and a discriminator, which are implemented as different neural networks. The generator generates a second image representing a change in the visual asset while the discriminator attempts to distinguish between the first image and the second image. The one or more processors update a first model in the discriminator and / or a second model in the generator based on whether the discriminator successfully distinguishes between the first and second images. Once trained, the generator generates images of the visual asset based on the first model, for example, based on a label or outline of the visual asset.

Description

Background Art

[0001] A significant portion of the budget and resources allocated to producing video games is consumed by the process of creating visual assets for them. For example, massively multiplayer online games include thousands of player avatars and non-player characters (NPCs), typically created using three-dimensional (3D) templates that are hand-customized during game development to create personalized characters. As another example, the environments or scenes of scenes in video games often include numerous virtual objects, such as trees, rocks, and clouds. These virtual objects are hand-customized to avoid excessive repetition or homogeneity, such as might occur when a forest contains hundreds of identical trees or a repeating pattern of trees. Procedural content generation has been used to generate characters and objects, but the content generation process is difficult to control and often produces visually uniform, homogeneous, or repetitive output. The high cost of producing visual assets for video games drives up video game budgets, which increases risk aversion among video game producers. Furthermore, the cost of content generation is a significant barrier to entry for smaller studios (with correspondingly smaller budgets) attempting to enter the high-fidelity game design market. Furthermore, video game players, especially online players, have come to expect frequent content updates, further exacerbating the problems associated with the high cost of producing visual assets. Summary of the Invention

[0002] The proposed solution particularly relates to a computer-implemented method comprising: capturing a first image of a three-dimensional (3D) digital representation of a visual asset; generating a second image representing a variation of the visual asset using a generator in a generative adversarial network (GAN) and attempting to distinguish between the first and second images in a discriminator in the GAN; updating at least one of a first model in the discriminator and a second model in the generator based on whether the discriminator successfully distinguishes between the first and second images; and generating a third image using the generator based on the updated second model. The first model is used by the generator as a basis for generating the second image, while the second model is used by the discriminator as a basis for evaluating the generated second image. The variation of the first image generated by the generator can, in particular, involve a variation of at least one image parameter of the first image, such as a variation of at least one or all pixel or texel values ​​of the first image. Thus, the variation by the generator can, for example, involve a variation of at least one of color, brightness, texture, granularity, or a combination thereof.

[0003] Machine learning has been used, for example, to generate images using neural networks trained on image databases. One image generation method used in the current context uses a machine learning architecture called a generative adversarial network (GAN), which learns how to create different types of images using pairs of interacting convolutional neural networks (CNNs). The first CNN (the generator) creates new images corresponding to images in a training dataset, and the second CNN (the discriminator) attempts to distinguish between the generated images and "real" images from the training dataset. In some cases, the generator produces images based on cues and / or random noise that guide the image generation process; in these cases, the GAN is referred to as a conditional GAN ​​(CGAN). Typically, a "cue" in the current context can be, for example, a parameter representing the image content in a computer-readable format. Examples of cues include labels associated with the image and shape information such as the outline of an animal or object. The generator and the discriminator then compete based on the images generated by the generator. The generator "wins" if the discriminator classifies the generated image as a real image (or vice versa), and the discriminator "wins" if the discriminator correctly classifies both the generated and real images. The generator and discriminator can update their respective models based on the loss function, which encodes the win or loss as the "distance" from the correct model. The generator and discriminator continue to improve their respective models based on the results produced by another CNN.

[0004] The generator in a trained GAN produces images that attempt to mimic the characteristics of people, animals, or objects in the training dataset. As described above, the generator in a trained GAN can generate images based on prompts. For example, in response to receiving a prompt containing the label "bear," a trained GAN will attempt to generate an image that resembles a bear. However, the images produced by a trained GAN are (at least in part) determined by the characteristics of the training dataset and may not reflect the intended characteristics of the generated images. For example, video game designers often create visual identities for their games using a fantasy or science fiction style characterized by dramatic perspective, image composition, and lighting effects. In contrast, traditional image databases include real-world photographs of a variety of different people, animals, or objects taken in different environments under varying lighting conditions. Furthermore, photographic face datasets are typically pre-processed to include a limited number of viewpoints, rotated to ensure faces are not tilted, and modified by applying a Gaussian blur to the background. Consequently, a GAN trained on a traditional image database will be unable to generate images that maintain the visual identity created by the game designer. For example, images that mimic people, animals, or objects from real-world photography can disrupt the visual coherence of scenes created in a fantasy or science fiction style. Furthermore, large repositories of illustrations that could otherwise be used for GAN training suffer from issues of ownership, style conflicts, or simply a lack of the diversity required to build robust machine learning models.

[0005] The proposed solution therefore provides a hybrid process pipeline for generating diverse and visually coherent content by training the generator and discriminator of a conditional generative adversarial network (CGAN) using images captured from three-dimensional (3D) digital representations of visual assets. The 3D digital representation includes a model of the visual asset's 3D structure and, in some cases, textures applied to the model's surface. For example, a 3D digital representation of a bear can be represented by a collection of triangles, other polygons, or patches, collectively referred to as primitives, and textures applied to the primitives to incorporate visual details, such as fur, teeth, claws, and eyes, at a higher resolution than the primitives. Training images ("first images") are captured using a virtual camera that captures the images from different viewpoints and, in some cases, under different lighting conditions. By capturing training images of 3D digital representations of visual assets, an improved training dataset can be provided, resulting in the generation of diverse and visually coherent content composed of a variety of second images of 3D representations of various visual assets that can be used individually, independently, or in combination within a video game. Capturing training images ("first images") via the virtual camera can include capturing a set of training images associated with different perspectives or lighting conditions of the 3D representation of the virtual asset. The number of training images in the training set, at least one of the perspectives, or lighting conditions, can be predetermined by a user or an image capture algorithm. For example, at least one of the number of training images in the training set, the perspective, and the lighting conditions can be preset or dependent on the visual asset for which the training images are to be captured. This can include, for example, automatically capturing the training images after the visual asset has been loaded into the image capture system and / or after the image capture process for implementing the virtual camera has been triggered.

[0006] The image capture system can also apply labels to the captured images, including labels indicating the type of object (e.g., bear), camera position, camera pose, lighting conditions, texture, and color. In some embodiments, the image is segmented into different parts of a visual asset, such as an animal's head, ears, neck, legs, and arms. The segmented portions of the image can be labeled to indicate the different parts of the visual asset. The labeled images can be stored in a training database.

[0007] By training a GAN, the generator and discriminator learn distributions of parameters representing images in a training database generated from 3D digital representations. That is, the GAN is trained using images from the training database. Initially, the discriminator is trained to identify “real” images of 3D digital representations based on images in the training database. The generator then begins generating a (second) image, for example, in response to a prompt such as a label or digital representation of the outline of a visual asset. The generator and discriminator can then iteratively and concurrently update their respective models, for example based on a loss function that indicates how well the generator is generating images representing the visual asset (e.g., how well it “fools” the discriminator) and how well the discriminator distinguishes the generated images from real images from the training database. The generator models the parameter distribution in the training images, and the discriminator models the parameter distribution inferred by the generator. Thus, the generator’s first model can include the parameter distribution in the first image, while the discriminator’s second model includes the parameter distribution inferred by the generator.

[0008] In some embodiments, the loss function includes a perceptual loss function that uses another neural network to extract features from the image and encodes the difference between the two images as the distance between the extracted features. In some embodiments, the loss function can receive a classification decision from the discriminator. The loss function can also receive information indicating the identity (or at least true or false status) of the second image provided to the discriminator. The loss function can then generate a classification error based on the received information. The classification error indicates how well the generator and discriminator achieve their respective goals.

[0009] Once trained, the GAN is used to generate images representing visual assets based on the parameter distribution inferred by the generator. In some embodiments, the images are generated in response to prompts. For example, a trained GAN can generate an image of a bear in response to receiving a prompt that includes the label "bear" or a representation of a bear silhouette. In some embodiments, the images are generated based on a synthesis of segmented portions of the visual asset. For example, a chimera can be generated by combining image segments representing different creatures (as indicated by the corresponding labels) (the head, body, legs, and tail of a dinosaur and the wings of a bat).

[0010] In some embodiments, at least one third image can be generated at a generator in the GAN based on the first model to represent a variation of the visual asset. Generating the at least one third image can then, for example, include generating the at least one third image based on at least one of a label associated with the visual asset or a digital representation of an outline of a portion of the visual asset. Alternatively or additionally, generating the at least one third image can include generating the at least one third image by combining at least one segment of the visual asset with at least one segment of another visual asset.

[0011] The proposed solution also relates to a system comprising: a memory configured to store a first image captured from a three-dimensional (3D) digital representation of a visual asset; and, at least one processor configured to implement a generative adversarial network (GAN) comprising a generator and a discriminator, the generator configured to generate a second image representing a change in the visual asset, for example, while the discriminator attempts to distinguish between the first image and the second image, and the at least one processor configured to update at least one of a first model in the discriminator and a second model in the generator based on whether the discriminator successfully distinguishes between the first and second images.

[0012] The proposed system may in particular be configured to implement embodiments of the proposed method. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art, by reference to the accompanying drawings. The use of the same reference numerals in different drawings indicates similar or identical items.

[0014] Figure 1 is a block diagram of a video game processing system implementing a hybrid procedural machine language (ML) pipeline for art development, according to some embodiments.

[0015] Figure 2 is a block diagram of a cloud-based system implementing a hybrid procedural ML pipeline for art development, according to some embodiments.

[0016] Figure 3 is a block diagram of an image capture system for capturing an image of a digital representation of a visual asset, according to some embodiments.

[0017] Figure 4 is a block diagram of an image of a visual asset and tag data representing the visual asset, according to some embodiments.

[0018] Figure 5 is a block diagram of a generative adversarial network (GAN) trained to generate varied images as visual assets, in accordance with some embodiments.

[0019] Figure 6 is a flowchart of a method of training a GAN to generate variations of an image of a visual asset, according to some embodiments.

[0020] Figure 7 Illustrated is the evolution of the distribution of true values ​​of parameters characterizing an image of a visual asset and the distribution of corresponding parameters generated by a generator in a GAN, according to some embodiments.

[0021] Figure 8is a block diagram of a portion of a GAN that has been trained to generate varied images as visual assets, in accordance with some embodiments.

[0022] Figure 9 is a flow chart of a method of generating variations of an image of a visual asset, according to some embodiments. DETAILED DESCRIPTION

[0023] Figure 1 is a block diagram of a video game processing system 100 that implements a hybrid procedural machine language (ML) pipeline for art development, according to some embodiments. The processing system 100 includes or has access to a system memory 105 or other storage element implemented using a non-transitory computer-readable medium such as dynamic random access memory (DRAM). However, some embodiments of the memory 105 are implemented using other types of memory, including static RAM (SRAM) and non-volatile RAM, among others. The processing system 100 also includes a bus 110 to support communications between entities implemented in the processing system 100, such as the memory 105. Some embodiments of the processing system 100 include other buses, bridges, switches, routers, and the like, which are not shown here for clarity. Figure 1 Not shown.

[0024] Processing system 100 includes a central processing unit (CPU) 115. Some embodiments of CPU 115 include multiple processing elements (referred to as "CPUs" for clarity) that execute instructions simultaneously or in parallel. Figure 1 105). Processing elements are referred to as processor cores, computing units, or other terms. CPU 115 is connected to bus 110 and communicates with memory 105 via bus 110. CPU 115 executes instructions, such as program code 120, stored in memory 105 and stores information, such as the results of the executed instructions, in memory 105. CPU 115 can also initiate graphics processing by issuing draw calls.

[0025] The input / output (I / O) engine 125 handles input or output operations associated with the display 130, which presents images or video on the screen 135. In the illustrated embodiment, the I / O engine 125 is connected to a game controller 140, which provides control signals to the I / O engine 125 in response to a user pressing one or more buttons on the game controller 140 or otherwise interacting with the game controller 140 (e.g., using motion detected by an accelerometer). The I / O engine 125 also provides signals to the game controller 140 to trigger responses in the game controller 140, such as vibrations and lights. In the illustrated embodiment, the I / O engine 125 reads information stored on an external storage element 145, which is implemented using non-transitory computer-readable media such as a compact disc (CD), a digital video disc (DVD), etc. The I / O engine 125 also writes information to the external storage element 145, such as the results of processing by the CPU 115. Some embodiments of the I / O engine 125 are coupled to other elements of the processing system 100, such as a keyboard, mouse, printer, external disk, etc. The I / O engine 125 is coupled to the bus 110 so that the I / O engine 125 communicates with the memory 105, the CPU 115, or other entities connected to the bus 110.

[0026] Processing system 100 includes a graphics processing unit (GPU) 150, which renders images for presentation on screen 135 of display 130, for example, by controlling the pixels that make up screen 135. For example, GPU 150 renders objects to generate pixel values ​​that are provided to display 130, which uses the pixel values ​​to display an image representing the rendered object. GPU 150 includes one or more processing elements, such as a compute unit array 155, that execute instructions concurrently or in parallel. Some embodiments of GPU 150 are used for general-purpose computing. In the illustrated embodiment, GPU 150 communicates with memory 105 (and other entities connected to bus 110) via bus 110. However, some embodiments of GPU 150 communicate with memory 105 via a direct connection or through other buses, bridges, switches, routers, and the like. GPU 150 executes instructions stored in memory 105 and stores information in memory 105, such as the results of the executed instructions. For example, memory 105 stores instructions representing program code 160 to be executed by GPU 150.

[0027] In the illustrated embodiment, the CPU 115 and GPU 150 execute corresponding program codes 120, 160 to implement a video game application. For example, user input received via the game controller 140 is processed by the CPU 115 to modify the state of the video game application. The CPU 115 then transmits a draw call to instruct the GPU 150 to render an image representing the state of the video game application for display on the screen 135 of the display 130. As discussed herein, the GPU 150 may also perform general computations related to video games, such as executing a physics engine or a machine learning algorithm.

[0028] The CPU 115 or GPU 150 also executes program code 165 to implement a hybrid procedural machine language (ML) pipeline for art development. The hybrid procedural ML pipeline includes a first portion that captures images 170 of three-dimensional (3D) digital representations of visual assets from different perspectives and, in some cases, under different lighting conditions. In some embodiments, the virtual camera captures first images or training images of the 3D digital representations of the visual assets from different perspectives and / or under different lighting conditions. Images 170 may be captured automatically by the virtual camera (i.e., based on an image capture algorithm included in program code 165). Images 170 captured by the first portion of the hybrid procedural ML pipeline (e.g., the portion including the model and the virtual camera) are stored in memory 105. The visual assets for which images 170 are captured may be user-generated (e.g., using a computer-aided design tool) and stored in memory 105.

[0029] The second portion of the hybrid procedural ML pipeline includes a generative adversarial network (GAN) represented by program code and associated data (such as model parameters) indicated by block 175. The GAN 175 includes a generator and a discriminator, which are implemented as different neural networks. The generator generates a second image representing a variation of the visual asset, and simultaneously the discriminator attempts to distinguish between the first image and the second image. Parameters defining the ML model in the discriminator or generator are updated based on whether the discriminator successfully distinguishes between the first and second images. The parameters defining the model implemented in the generator determine the distribution of the parameters across the training images 170. The parameters defining the model implemented in the discriminator determine the distribution of the parameters inferred by the generator, e.g., based on the generator's model.

[0030] GAN 175 is trained to generate different versions of a visual asset based on cues or random noise provided to the trained GAN 175, in which case the trained GAN 175 can be referred to as a conditional GAN. For example, if GAN 175 is being trained based on a collection of images 170 of digital representations of red dragons, the generator in GAN 175 generates images representing variations of red dragons (e.g., blue dragons, green dragons, larger dragons, smaller dragons, etc.). The images generated by the generator, or training images 170, are selectively provided to the discriminator (e.g., by randomly selecting between training images 170 and generated images), and the discriminator attempts to distinguish between "real" training images 170 and "fake" images generated by the generator. The parameters of the models implemented in the generator and discriminator are then updated based on a loss function, the value of which is determined based on whether the discriminator successfully distinguishes between real and fake images. In some embodiments, the loss function also includes a perceptual loss function, which uses another neural network to extract features from the real and fake images and encodes the difference between the two images as the distance between the extracted features.

[0031] Once trained, the generator in GAN 175 produces variations of the training images, which are used to generate images or animations for video games. Figure 1 The processing system 100 shown in FIG performs image capture, GAN model training, and subsequent image generation using the trained model, but in some embodiments other processing systems are used to perform these operations. For example, a first processing system (in a manner similar to Figure 1 ) can perform image capture and store images of visual assets in memory accessible to a second processing system or transmit the images to the second processing system. The second processing system can perform model training for the GAN 175 and store parameters defining the trained model in memory accessible to a third processing system or transmit the parameters to the third processing system. The third processing system can then be used to generate images or animations for a video game using the trained model.

[0032] Figure 2 is a block diagram of a cloud-based system 200 implementing a hybrid procedural ML pipeline for art development according to some embodiments. The cloud-based system 200 includes a server 205 interconnected with a network 210. Figure 2While a single server 205 is shown in FIG, some embodiments of the cloud-based system 200 include more than one server connected to a network 210. In the illustrated embodiment, the server 205 includes a transceiver 215 that transmits signals to and receives signals from the network 210. The transceiver 215 can be implemented using one or more separate transmitters and receivers. The server 205 also includes one or more processors 220 and one or more memories 225. The processor 220 executes instructions, such as program code, stored in the memory 225, and the processor 220 stores information, such as results of the executed instructions, in the memory 225.

[0033] The cloud-based system 200 includes one or more processing devices 230, such as computers, set-top boxes, and game consoles, connected to a server 205 via a network 210. In the illustrated embodiment, the processing device 230 includes a transceiver 235 that transmits signals to and receives signals from the network 210. The transceiver 235 can be implemented using one or more separate transmitters and receivers. The processing device 230 also includes one or more processors 240 and one or more memories 245. The processors 240 execute instructions, such as program code stored in the memories 245, and the processors 240 store information in the memories 245, such as the results of the executed instructions. The transceiver 235 is connected to a display 250 that displays images or video on a screen 255, a game controller 260, and other text or voice input devices. Some embodiments of the cloud-based system 200 are therefore used by cloud-based game streaming applications.

[0034] Processor 220, processor 240, or a combination thereof executes program code to perform image capture, GAN model training, and subsequent image generation using the trained model. The division of labor between processor 220 in server 205 and processor 240 in processing device 230 varies in different embodiments. For example, server 205 may train a GAN using images captured by a remote video capture processing system and provide parameters defining the model in the trained GAN to processor 220 via transceivers 215, 235. Processor 220 may then use the trained GAN to generate images or animations that are variations of the visual assets used to capture the training images.

[0035] Figure 3 is a block diagram of an image capture system 300 for capturing an image of a digital representation of a visual asset according to some embodiments. The image capture system 300 is a block diagram of an image capture system 300 for capturing an image of a digital representation of a visual asset using Figure 1 The processing system 100 and Figure 2 Some embodiments of the processing system 200 shown are implemented.

[0036] Image capture system 300 includes a controller 305 implemented using one or more processors, memories, or other circuits. Controller 305 is connected to virtual camera 310 and virtual light source 315. Figure 3 For clarity, not all connections are shown. The image capture system 300 is used to capture images of a visual asset 320 represented as a digital 3D model. In some embodiments, the 3D digital representation of the visual asset 320 (in this example, a dragon) is represented by a collection of triangles, other polygons, or patches, collectively referred to as primitives, and textures applied to the primitives to include visual details with a higher resolution than the primitive resolution, such as the texture and color of the dragon's head, claws, wings, teeth, eyes, and tail. The controller 305 selects a position, orientation, or pose of the virtual camera 310, such as Figure 3 Controller 305 also selects the light intensity, direction, color, and other properties of the light generated by virtual light source 315 to illuminate visual asset 320. Different light characteristics or properties are used for different exposures of virtual camera 310 to generate different images of visual asset 320. The selection of the position, orientation, or pose of virtual camera 310 and / or the selection of the light intensity, direction, color, and other properties of the light generated by virtual light source 315 can be based on user selection or can be automatically determined by an image capture algorithm executed by image capture system 300.

[0037] The controller 305 tags the images (e.g., by generating metadata associated with the images) and stores them as tagged images 325. In some embodiments, the images are tagged with metadata that indicates the type of visual asset 320 (e.g., dragon), the position of the virtual camera 310 when the image was acquired, the pose of the virtual camera 310 when the image was acquired, the lighting conditions produced by the light source 315, the texture applied to the visual asset 320, and the color of the visual asset 320. In some embodiments, the images are segmented into different portions of the visual asset 320 that may vary during the proposed art development process, such as the head, claws, wings, teeth, eyes, and tail of the visual asset 320. The segmented portions of the image are labeled to indicate the different portions of the visual asset 320.

[0038] Figure 4 is a block diagram of an image 400 of a visual asset and labeled data 405 representing the visual asset according to some embodiments. The image 400 and the labeled data 405 are Figure 3Some embodiments of the image capture system 300 shown are generated. In the illustrated embodiment, image 400 is an image of a visual asset including a bird in flight. Image 400 is segmented into different parts, including a head 410, a beak 415, wings 420, 421, a body 425, and a tail 430. Labeled data 405 includes image 405 and an associated label "bird." Labeled data 405 also includes the segmented parts of image 405 and associated labels. For example, labeled data 405 includes image portion 410 and an associated label "head," image portion 415 and an associated label "beak," image portion 420 and an associated label "wings," image portion 421 and an associated label "wings," image portion 425 and an associated label "body," and image portion 430 and an associated label "tail."

[0039] In some embodiments, image portions 410, 415, 420, 421, 425, and 430 are used to train a GAN to create corresponding portions of other visual assets. For example, image portion 410 is used to train a GAN generator to create the "head" of another visual asset. Training the GAN using image portion 410 is performed in conjunction with training the GAN using other image portions corresponding to the "heads" of one or more other visual assets.

[0040] Figure 5 is a block diagram of a GAN 500 trained to generate images of variations as visual assets, according to some embodiments. The GAN 500 is implemented in Figure 1 The processing system 100 and Figure 2 In some embodiments of the cloud-based system 200 shown.

[0041] GAN 500 includes a generator 505 implemented using a neural network 510 that generates images based on a parameterized model distribution. Some embodiments of generator 505 generate images based on input information such as random noise 515 and cues 520 in the form of labels or silhouettes of visual assets. GAN 500 also includes a discriminator 525 implemented using a neural network 530 that attempts to distinguish between images generated by generator 505 and labeled images 535 of visual assets, the latter representing true-value images. Thus, discriminator 525 receives either an image generated by generator 505 or a labeled image 535 and outputs a classification decision 540 indicating whether discriminator 525 believes the received image is a (fake) image generated by generator 505 or a (real) image from a set of labeled images 535.

[0042] The loss function 545 receives the classification decision 540 from the discriminator 525. The loss function 545 also receives information indicating the identity (or at least the real or fake status) of the corresponding image provided to the discriminator 525. The loss function 545 then generates a classification error based on the received information. The classification error indicates how well the generator 505 and the discriminator 525 achieve their respective goals. In the illustrated embodiment, the loss function 545 also includes a perceptual loss function 550, which extracts features from the real image and the fake image and encodes the difference between the real image and the fake image as the distance between the extracted features. The perceptual loss function 550 is implemented using a neural network 555 trained based on the labeled image 535 and the images generated by the generator 505. The perceptual loss function 550 thus contributes to the overall loss function 545.

[0043] The goal of the generator 505 is to deceive the discriminator 525, that is, to make the discriminator 525 identify the (fake) generated image as a (real) image drawn from the labeled image 535, or to identify the real image as a fake image. The model parameters of the neural network 510 are therefore trained to maximize the classification error (between real images and fake images) represented by the loss function 545. The goal of the discriminator 525 is to correctly distinguish between real images and fake images. The model parameters of the neural network 530 are therefore trained to minimize the classification error represented by the loss function 545. The training of the generator 505 and the discriminator 525 is performed iteratively and the parameters defining their corresponding models are updated during each iteration. In some embodiments, the gradient ascent method is used to update the parameters of the model implemented in the generator 505, thereby increasing the classification error. The gradient descent method is used to update the parameters of the model implemented in the discriminator 525, thereby reducing the classification error.

[0044] Figure 6 is a flow chart of a method 600 for training a GAN to generate variations of an image of a visual asset according to some embodiments. The method 600 is implemented in Figure 1 The processing system 100 shown, Figure 2 The cloud-based system 200 and Figure 5 In some embodiments of GAN500 shown in FIG.

[0045] At block 605, a first neural network implemented in a discriminator of a GAN is initially trained to recognize images of a visual asset using a collection of labeled images captured from the visual asset. Figure 3 The image capture system 300 is shown capturing some embodiments of tagged images.

[0046] At block 610, a second neural network implemented in the generator of the GAN generates an image representing a variation of the visual asset. In some embodiments, the image is generated based on input random noise, cues, or other information. At block 615, the generated image or an image selected from a set of labeled images is provided to the discriminator. In some embodiments, the GAN randomly selects between the (fake) generated image and the (true) labeled image provided to the discriminator.

[0047] At decision block 620, the discriminator attempts to distinguish between real images and fake images received from the generator. The discriminator makes a classification decision indicating whether the discriminator identifies the image as real or fake and provides the classification decision to a loss function, which determines whether the discriminator correctly identifies the image as real or fake. If the classification decision from the discriminator is correct, method 600 flows to block 625. If the classification decision from the discriminator is incorrect, method 600 flows to block 630.

[0048] At block 625, the model parameters defining the model distribution used by the first neural network in the generator are updated to reflect the fact that the images generated by the generator did not successfully deceive the discriminator. At block 630, the model parameters defining the model distribution used by the second neural network and the discriminator are updated to reflect the fact that the discriminator did not correctly identify whether the received image was real or fake. Figure 6 The method 600 shown in depicts the model parameters at the generator and discriminator being updated independently, but some embodiments of the GAN update the model parameters of the generator and discriminator simultaneously based on a loss function determined in response to the discriminator providing a classification decision.

[0049] At decision block 635, the GAN determines whether the training of the generator and the discriminator has converged. Convergence is assessed based on the magnitude of change in the parameters of the models implemented in the first and second neural networks, the fractional change in the parameters, the rate of change of the parameters, a combination thereof, or based on other criteria. If the GAN determines that the training has converged, the method 600 proceeds to block 640 and the method 600 ends. If the GAN determines that the training has not converged, the method 600 proceeds to block 610 and performs another iteration. Although each iteration of the method 600 is performed for a single (real or fake) image, some embodiments of the method 600 provide multiple real and fake images to the discriminator in each iteration and then update the loss function and model parameters based on the classification decisions returned by the discriminator for the multiple images.

[0050] Figure 7 The diagram illustrates the evolution of the distribution of true values ​​of parameters representing an image of a visual asset and the distribution of corresponding parameters generated by the generator in a GAN according to some embodiments. Figure 6In the illustrated method 600, the distribution is presented in three consecutive time intervals 701, 702, and 703, which correspond to consecutive iterations of training the GAN. The values ​​of the parameters corresponding to the labeled images (true images) captured from the visual asset are indicated by open circles 705, and for clarity, only one is indicated by a reference number in each of the time intervals 701-703.

[0051] During the first time interval 701, the parameter values ​​corresponding to the images (fake images) generated by the generator in the GAN are indicated by solid circles 710, with only one indicated by a reference numeral for clarity. The distribution of parameters 710 for the fake images is significantly different from the distribution of parameters 705 for the real images. Therefore, the discriminator in the GAN has a high probability of successfully distinguishing between real and fake images during the first time interval 701. Therefore, the neural network implemented in the generator is updated to improve its ability to generate fake images that deceive the discriminator.

[0052] During the second time interval 702, the values ​​of the parameters corresponding to the images generated by the generator are indicated by filled circles 715, with only one indicated by a reference numeral for clarity. The distribution of the parameters 715 representing the fake images is more similar to the distribution of the parameters 705 representing the real images, indicating that the neural network in the generator was successfully trained. However, the distribution of the parameters 715 for the fake images is still significantly different from the distribution of the parameters 705 for the real images (although the difference is smaller). Therefore, during the second time interval 702, the discriminator in the GAN is likely to successfully distinguish between real and fake images. The neural network implemented in the generator is again updated to improve its ability to generate fake images for the discriminator.

[0053] During the third time interval 703, the parameter values ​​corresponding to the images generated by the generator are indicated by filled circles 720, with only one indicated by a reference numeral for clarity. The distribution of parameters 720 representing fake images is now barely distinguishable from the distribution of parameters 705 representing real images, indicating that the neural network in the generator is being successfully trained. Therefore, during the third time interval 703, the discriminator in the GAN has a low probability of successfully distinguishing between real and fake images. Consequently, the neural network implemented in the generator has converged on a model distribution for generating variations of the visual asset.

[0054] Figure 8 is a block diagram of a portion 800 of a GAN that has been trained to generate images that are variations of a visual asset, according to some embodiments. Figure 1 The processing system 100 and Figure 2200 . The GAN portion 800 includes a generator 805 implemented using a neural network 810 that generates images based on a model distribution of parameters. As discussed herein, the model distribution of parameters has been trained based on a collection of labeled images captured of visual assets. The trained neural network 810 is used to generate images or animations 815 representing changes in the visual asset, such as for use in a video game. Some embodiments of the generator 805 generate images based on input information such as random noise 820 and a hint 825 in the form of a label or outline of the visual asset.

[0055] Figure 9 is a flow chart of a method 900 for generating a variation of an image of a visual asset according to some embodiments. The method 900 is implemented in Figure 1 The processing system 100 shown, Figure 2 The cloud-based system 200 shown, Figure 5 GAN 500 and Figure 8 In some embodiments of the portion 800 of the GAN shown in FIG.

[0056] At block 905, a hint is provided to the generator. In some embodiments, the hint is a digital representation of a sketch of a portion (e.g., a silhouette) of the visual asset. The hint may also include tags or metadata used to generate the image. For example, the tag may indicate the type of visual asset, such as "dragon" or "tree." For another example, if the visual asset is segmented, the tag may indicate one or more segments.

[0057] At block 910, random noise is provided to the generator. Random noise can be used to add a degree of randomness to the variations in the images produced by the generator. In some embodiments, both the prompt and the random noise are provided to the generator. However, in other embodiments, one or the other of the prompt of the random noise is provided to the generator.

[0058] At block 915, the generator generates images representing variations of the visual asset based on the prompt, random noise, or a combination thereof. For example, if the label indicates the type of visual asset, the generator generates images representing variations of the visual asset using images with the corresponding label. For another example, if the label indicates a segment of the visual asset, the generator generates images representing variations of the visual asset based on images of the segment with the corresponding label. Thus, multiple variations of a visual asset can be created by combining differently labeled images or segments. For example, a chimera can be created by combining the head of one animal with the body of another animal and the wings of a third animal.

[0059] In some embodiments, certain aspects of the above-described technology can be implemented by one or more processors of a processing system that executes software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions and certain data that, when executed by one or more processors, manipulate the one or more processors to perform one or more aspects of the above-described technology. The non-volatile computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as flash memory, a cache, a random access memory (RAM), or other one or more non-volatile storage devices, etc. The executable instructions stored on the non-transitory computer-readable storage medium may be in source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executable by one or more processors.

[0060] Computer-readable storage media may include any storage medium or combination of storage media that is accessible by a computer system during use for providing instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache memory), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical systems (MEMS)-based storage media. Computer-readable storage media may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disc or flash memory based on a universal serial bus (USB), or coupled to a computer system via a wired or wireless network (e.g., a network accessible storage (NAS)).

[0061] Note that not all of the activities or elements described above in the general description are required, a portion of a particular activity or device may not be required, and one or more other activities may be performed or elements may be included in addition to those described above. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Moreover, the concepts have been described with reference to specific embodiments. However, it will be understood by those skilled in the art that various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the appended claims. Accordingly, the specification and drawings should be regarded as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the present disclosure.

[0062] Benefits, other advantages and solutions to problems have been described above with respect to specific embodiments. However, benefits, advantages, solutions to problems, and any features that may cause any benefit, advantage, or solution to a problem to appear or become more apparent should not be construed as key, required, or essential features of any or all of the claims. Furthermore, the specific embodiments disclosed above are illustrative only, as the disclosed subject matter may be modified and practiced in different but equivalent manners that are apparent to those skilled in the art having the benefit of the teachings herein. There is no intention to be limited to the details of construction or design shown herein, except as described in the claims below. It is therefore apparent that the specific embodiments disclosed above may be altered or modified, and all such variations are considered to be within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as described in the appended claims.

Claims

1. A computer-implemented method comprising: capturing a first image of a three-dimensional 3D digital representation of a visual asset; generating a second image representing a variation of the visual asset using a generator in a generative adversarial network (GAN), and attempting to distinguish between the first image and the second image at a discriminator in the GAN; updating at least one of a first model in the discriminator and a second model in the generator based on whether the discriminator successfully distinguishes between the first image and the second image; as well as generating a third image based on the updated second model using the generator, The 3D digital representation includes primitives and textures, the primitives being composed of polygons or patches, and the textures being applied to the primitives to include visual details having a higher resolution than a resolution of the primitives.

2. The method according to claim 1, wherein Capturing the first image from the 3D digital representation of the visual asset includes capturing the first image using a virtual camera that captures the first image from different viewing angles and under different lighting conditions.

3. The method according to claim 2, wherein: Capturing the first image includes tagging the first image based on at least one of a type of the visual asset, a position of the virtual camera, a pose of the virtual camera, a texture applied to the visual asset, and a color of the visual asset.

4. The method according to claim 3, wherein: Capturing the first image includes segmenting the first image into portions associated with different portions of the visual asset and marking the portions of the first image to indicate the different portions of the visual asset.

5. The method according to claim 4, wherein Generating a third image based on the updated second model using the generator includes: At the generator in the GAN, at least one third image representing a change of the visual asset is generated based on the updated second model by combining at least one labeled portion of the visual asset with at least one labeled portion of another visual asset.

6. The method according to claim 3, wherein: Updating at least one of the first model and the second model includes applying a loss function that indicates at least one of a first likelihood that the discriminator cannot distinguish the second image from the first image and a second likelihood that the discriminator successfully distinguishes the first image from the second image.

7. The method according to claim 6, wherein: The first model comprises a first distribution of parameters in the first image, and wherein the second model comprises a second distribution of parameters inferred by the generator.

8. The method according to claim 7, wherein: Applying the loss function includes applying a perceptual loss function that extracts features from the first image and the second image and encodes differences between the first image and the second image as distances between the extracted features.

9. The method according to any one of claims 3 to 8, further comprising: At least one third image is generated at the generator in the GAN based on the first model to represent a change in the visual asset.

10. The method according to claim 9, wherein: Generating the at least one third image includes generating the at least one third image based on at least one of a tag associated with the visual asset or a digital representation of an outline of a portion of the visual asset.

11. The method according to claim 9, wherein Generating the at least one third image includes generating the at least one third image by combining at least a portion of the visual asset with at least a portion of another visual asset.

12. A computer program product comprising a set of executable instructions for manipulating at least one processor to perform the method according to any one of claims 1 to 11.

13. A system comprising: a memory configured to store a set of executable instructions; as well as At least one processor is configured to execute the set of executable instructions to perform the method according to any one of claims 1 to 11.

Images