A method and platform for generating anime-style game backgrounds based on generative adversarial networks

Abstract

This invention discloses a method and platform for generating anime-style game backgrounds based on generative adversarial networks (GANs). The method includes constructing a game background image, obtaining a dataset from the game background image, constructing a color transfer network for generating color segmentation maps and a style reconstruction network for reconstructing anime-style maps, creating a GAN for anime style transfer using Python in the model preparation stage, generating line images corresponding to training images using an edge extraction algorithm, extracting color segmentation images from images in the dataset using a color segmentation algorithm, and building network models G1 and G2 for the color transfer and style reconstruction stages using Python programming language to generate the target anime-style game background image C2.

Description

Technical Field

[0001] This invention relates to the field of robot control, and more specifically, to a method and platform for generating anime-style game backgrounds based on generative adversarial networks. Background Technology

[0002] In the development of electronic information products, video games, as a form of entertainment based on digital devices, are becoming increasingly popular, boasting a huge market and audience worldwide. Simultaneously, the promotion of games also stimulates research in related scientific fields, such as computer vision and artificial intelligence. Anime-style video games, such as visual novels and role-playing games, typically feature backgrounds with varying time states. Through the progression of the plot, character portraits, and scene transitions, players are immersed in the game's world. However, high-quality background rendering in video game development faces the following challenges:

[0003] (1) Limited creativity. Like other cultural products, unique creativity and inspiration are needed to stand out from numerous game projects.

[0004] (2) Insufficient resources. Creating a game background image, from photography and sketching to coloring and correction, often requires a large amount of human and material resources. In addition, training personnel who can create game backgrounds requires a significant investment of resources and takes a long time.

[0005] Therefore, how to utilize the development of computer technology to address the aforementioned shortcomings has become an urgent problem to be solved.

[0006] Existing technologies do not perform style transfer for anime images used in games. For example, Chinese invention patent application CN113989106A discloses an arbitrary aesthetic style transfer method. However, it requires two structurally similar images and the generated images cannot maintain clear edges. It works well for style transfer of Impressionist and Abstract styles, but its effect is not ideal for style transfer of anime styles that require clear edges. Summary of the Invention

[0007] To address at least one of the aforementioned technical problems, this invention proposes a method and platform for generating anime-style game backgrounds based on generative adversarial networks.

[0008] The first aspect of this invention provides a method for generating anime-style game backgrounds based on generative adversarial networks, comprising the following steps:

[0009] S1, Construct game background images. Obtain the dataset from the game background images. Each animation scene in the dataset consists of images from three different time points. Label each image with attribute label i.

[0010] S2, Construct a color transfer network for generating color segmentation maps and a style reconstruction network for reconstructing anime style maps. Use a pre-trained VGG network and Gram matrix to construct a loss function to optimize the structure and style of the generated images.

[0011] The VGG network consists of 5 convolutional layers, 3 fully connected layers, and a softmax output layer. All hidden layer activation units use the ReLU function. Using a Gram matrix to construct the loss function means representing image features by calculating the inner product of feature maps between channels.

[0012] S3: In the model preparation stage, a generative adversarial network (GAN) for anime style transfer was created using Python. In the input stage, edge detection and image segmentation algorithms were used to generate line drawings and color block images of the photos, which were then combined with anime-style images for unsupervised training. A conditional generative adversarial network was used to perform color transfer on the color segmentation images of the photos. The color block images were combined with the target time node labels and input into the generator to generate images at different time nodes. Unsupervised training refers to using unpaired data during training, which has the advantage of increasing sample diversity.

[0013] S4, use the edge extraction algorithm to generate line images corresponding to the training images;

[0014] S5 uses a color segmentation algorithm to extract color segmentation images from the images in the dataset;

[0015] S6. Use Python programming language to build network models G1 and G2 for the color transfer stage and style reconstruction stage. Train G1 and G2 with the collected dataset. Input the photo P and time node label i into generator G1 to generate color segmentation image C1 corresponding to label i. Then input the edge detection image E(C1) of C1 and P into generator G2 to generate target anime style game background image C2.

[0016] In a preferred embodiment of the present invention, in step S1, the attribute tag i takes the values ​​0, 1, and 2. When the attribute tag value is 0, it means that the time of the image is daytime. When the attribute tag value is 1, it means that the time of the image is dusk. When the attribute tag value is 2, it means that the time of the image is nighttime.

[0017] In a preferred embodiment of the present invention, the image with a data concentration ratio of 16:9 in step S1 is processed as follows;

[0018] The image with a 16:9 aspect ratio in the dataset is cropped in the middle to obtain two 9:8 images, which are then resized to 512×512 pixels. Finally, the resulting 512×512 images are cropped again along the halfway point of the x and y axes to create four 256×256 pixels.

[0019] In a preferred embodiment of the present invention, the method for generating line images corresponding to training images using an edge extraction algorithm in step S4 is as follows: The total number of scenes in the game scene database is M, C(x t (x, i) represents the background image of the t-th scene at time i, where t ranges from [1, M] and i is any one of 0, 1, or 2. For each scene, an image from a daytime time node is selected as the base image, and an edge detection image corresponding to this scene is generated. If the training image is C(x... t If i), then the corresponding line graph is E(C(x). t ,i)).

[0020] In a preferred embodiment of the present invention, a color transfer network loss function is constructed, which consists of two parts: an adversarial loss function. Content loss function: Where G1 refers to the generator in the color transfer stage, and D1 refers to the discriminator in the color transfer stage. The constructed adversarial loss function is:

[0021]

[0022] In the formula, S represents the Felzenszwalb image segmentation algorithm, log refers to the logarithmic function, and E... y ~M refers to sampling an image y from dataset M, E x ~P refers to sampling an image x from dataset P. The constructed content loss function is:

[0023]

[0024] The anime image dataset is M, where y is one of the anime images, and t is the corresponding time node label, with values ​​of 0.0, 0.5, and 1.0 representing daytime, dusk, and nighttime, respectively. The photo dataset is P, where x is one of the photos, S represents the color segmentation algorithm, and F... gray This is a commonly used formula in video engineering for converting color to grayscale, where ||1 represents the root mean square error, E x～P This refers to sampling an image x from a dataset P, where VGG represents a pre-trained VGG network, and x... r x g x b If we consider the red, green, and blue channels of the image respectively, then:

[0025] F gray (x) = 0.299*x r +0.588*x g +0.144*x b .

[0026] In a preferred embodiment of the present invention, the loss function of the style reconstruction network is constructed, including an adversarial loss function. Content loss function Style loss function With color segmentation loss function

[0027] In a preferred embodiment of the present invention, the adversarial loss function is:

[0028]

[0029] Among them, F edge It is the Laplacian edge extraction operator, F fel In the Felzenszwalb image segmentation algorithm, G2 refers to the generator in the style reconstruction stage, D2 refers to the discriminator in the style reconstruction stage, X' refers to the image generated in the color transfer stage, and E... y～M This refers to sampling an image y from a dataset M, E x～P This refers to sampling an image x from dataset P.

[0030] The constructed content loss function is:

[0031]

[0032] Here, VGG refers to a pre-trained VGG network, F gray This refers to the formula for converting color to grayscale, where ||1 represents the root mean square error.

[0033] The constructed style loss function is:

[0034]

[0035] The constructed color segmentation loss function is:

[0036]

[0037] Here, Gram represents the Gram matrix.

[0038] In a preferred embodiment of the present invention, generators G1 and G2 include two parts: encoding and decoding. The encoding part includes a fully connected layer, a downsampling layer, and multiple residual blocks, each of which is a skip connection. The decoding part includes an upsampling layer and a fully connected layer. Each downsampling by the generator reduces the size of each feature map to half the size of the previous feature map, and each upsampling reduces the size of the feature map to twice the size of the previous feature map.

[0039] The second aspect of this invention provides an anime-style game background generation platform based on generative adversarial networks. The platform includes a browser and a server. The server generates game backgrounds using an anime-style game background generation method. When a user uses the platform, they directly call a training model stored in the server's backend to train the game background. The browser is used to display the game background.

[0040] The technical solution of the present invention has the following advantages compared with the prior art:

[0041] 1. With the widespread use of the internet, almost every electronic device has a web browser, and most browsers support running JavaScript. Users don't need to configure a complicated environment; all they need is a computer with a browser to create and generate game backgrounds. The operation is flexible and simple.

[0042] 2. It can effectively convert input images into anime-style video game background images while maintaining clear edges, and generate difference maps under different time conditions.

[0043] 3. It has a user-friendly interface. The web platform application is based on familiar user interaction logic, so that users no longer have any obstacles in calling the model. In addition, this application adds a content loss function to ensure the structural consistency between the generated result and the input image, and adds a style loss function to optimize the network output.

[0044] Deep learning can be used to solve image segmentation and classification tasks, extract high-dimensional features of images, and obtain high-quality output. Generative adversarial networks can be used to train and generate new image data from existing sample data, improving the speed and accuracy of sample data training. Attached Figure Description

[0045] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, some of the drawings in the following description are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1This is a flowchart of the method for generating anime-style game backgrounds in a preferred embodiment of the present invention;

[0047] Figure 2 This is an evaluation comparison table obtained using FID and user evaluation metrics in a preferred embodiment of the present invention. Detailed Implementation

[0048] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0049] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0050] Example 1:

[0051] like Figure 1-2 As shown, the present invention provides a method for generating anime-style game backgrounds based on generative adversarial networks, comprising two generative adversarial networks, consisting of a generator G1 for color transfer, a discriminator D1, and a generator G2 and a discriminator D2 for style reconstruction. G1 and D1 are trained before G2 and D2.

[0052] Edge detection algorithms are used to solve fundamental edge detection problems in image processing, such as the Canny operator using traditional filters and nested edge detection based on deep learning convolutional networks. Color segmentation algorithms are used to merge pixels with similar properties. Anime images generally have clear edges and smooth textures, and in the process of hand-drawing, drawing is often divided into line art and background color stages. An improved conditional generative adversarial network (GAN) game background stylization algorithm uses edge detection and color segmentation algorithms to extract line art and color segmentation maps from images to simulate actual hand-drawn line art and background colors. The Laplacian edge detection operator, which has strong and weak edge details, and the clustering-based Felzenszwalb algorithm are used to preprocess the training data.

[0053] G1 is used to convert the colors of the color segmentation map into specific time-node feature colors. Photo x1 is processed using the Felzenszwalb image segmentation algorithm to generate a color segmentation map x′1, which is then concatenated with the time label t and input into generator G1. G1 outputs an image y1 whose colors match the semantic representation of label t. Discriminator D1 determines whether the input segmentation map is a color segmentation map of an anime image and whether it matches the specific time-node semantic features. G2 encodes and decodes the features of the input line drawing and color segmentation map into a complete anime image. Photo x2 is input into the trained G1 to obtain a color block image with specific time-node feature colors. Edges of x2 are then extracted using an edge detection algorithm to generate a line drawing x′2, which is concatenated with the color image and input into G2 for training. D2 determines whether the input image is an anime image.

[0054] The loss function of the color transfer network consists of two parts: adversarial loss and content loss. set up Given the final weights of the generator network, the goal of the color transfer network is:

[0055]

[0056] λ1 and λ2 represent the weights of the adversarial loss function and the content loss function. Setting λ1 to 10 and λ2 to 0.1 can achieve relatively good results.

[0057] Adversarial loss function Applied to both the generator and discriminator, its value indicates how closely the color segmentation map of the G1 output image resembles the color segmentation map of the anime image, and how closely it corresponds to the time node label. Let the anime image dataset be M, y be one of the anime images, and t be the corresponding time node label, where t takes the value of 0.0, 0.5, or 1.0, representing daytime, dusk, and nighttime, respectively. Let the photo dataset be P, x be one of the photos, and S represent the color segmentation algorithm. Define the adversarial loss function. The expression is:

[0058]

[0059] Using only adversarial loss functions can cause the output color segmentation map to lose the original image segmentation structure. Our proposed method uses a structural loss function to constrain the consistency of the generated image with the input image in terms of structural semantics. In this method, a pre-trained VGG network is used to extract high-dimensional semantic features of the image, and the L1 loss function of the generated image and the original image in terms of structural semantics is calculated. Since only structural consistency with the input is required, and color consistency is not necessary, the output color map needs to be converted to grayscale. Let F... grayThe formula for converting color to grayscale, commonly used in video engineering, is x. r x g x b If we consider the red, green, and blue channels of the image respectively, then:

[0060] F gray (x) = 0.299*x r +0.588*x g +0.144*x b

[0061] The content loss function is:

[0062]

[0063] The loss function of the style reconstruction network consists of four parts: adversarial loss, content loss, style loss, and color segmentation loss, as follows: set up Given the final weights of the generator network, the goal of the style reconstruction network is:

[0064]

[0065] γ1, γ2, γ3, and γ4 represent the weights of the adversarial loss function, content loss function, style loss function, and color segmentation loss function.

[0066] The value indicates how closely the output image resembles the anime image, x′ is the color segmentation map transformed by the generator G1 trained during the color transfer stage, and F edge It is the Laplacian edge extraction operator, F fel If it is the Felzenszwalb image segmentation algorithm, then:

[0067]

[0068] Content loss functions are used to preserve the structure of the input image in the generated image, and both the input and output need to be converted to grayscale.

[0069]

[0070] The content loss function is used to give the generated image an anime style, and the style loss function is constructed using a Gram matrix:

[0071]

[0072] The color segmentation loss is used to ensure that the generated image retains the colors of the color segmentation map generated in the G1 stage, maintaining consistency with the temporal label semantics of the input G1:

[0073]

[0074] We construct a dataset of anime background images labeled with time information. Each anime scene in the dataset consists of images from three different time points. We build a color transfer network for generating color segmentation maps and a style reconstruction network for reconstructing anime style maps. We use a pre-trained VGG network and Gram matrix to construct a loss function to optimize the structure and style of the generated images.

[0075] The framework of this invention consists of two generative adversarial networks: a generator G1 and a discriminator D1 for color transfer, and a generator G2 and a discriminator D2 for style reconstruction. G1 and D1 are trained before G2 and D2. G1 converts the colors of a color segmentation map into specific time-node feature colors. A photo x1 is processed using the Felzenszwalb image segmentation algorithm to generate a color segmentation map x′1. x′1 is concatenated with a time label t and input into generator G1. G1 outputs an image y′1 whose colors match the semantic representation of label t. Discriminator D1 determines whether the input segmentation map is a color segmentation map of an anime image and whether it matches the specific time-node semantic features. G2 encodes and decodes the features of the input line drawing and color segmentation map into six complete anime images. A photo x2 is input into the trained G1 to obtain a color block image with specific time-node feature colors. Edges of x2 are then extracted using an edge detection algorithm to generate a line drawing x′2, which is concatenated with the color image and input into G2 for training. D2 determines whether the input image is an anime image.

[0076] The generators and discriminators of the two adversarial generative networks in this paper have the same structure. In the figure, f represents the number of feature maps output by each convolutional layer, k represents the size of the convolutional kernel, and s represents the stride. The input image G1 has 4 channels: the red, green, and blue channels of the color block image and the word embedding layer of the time label. The input image G2 also has 4 channels: the line image channel and the red, green, and blue channels of the color block image output by G1. Generators G1 and G2 are divided into encoding and decoding parts. The encoding part consists of a fully connected layer, two downsampling layers, and eight residual blocks. To prevent gradient vanishing, each residual block has a skip connection. The decoding part consists of two upsampling layers and a fully connected layer. Each downsampling by the generator reduces the size of each feature map to half the size of the previous feature map, and each upsampling increases the size of the feature map to twice the size of the previous feature map. Discriminators D1 and D2 consist of two fully connected layers and two distributed downsampling layers. The last fully connected layer outputs a single-channel image that is one-quarter the size of the original image. The value of each pixel represents the judgment value for a receptive field region. Finally, the values ​​of all pixels are averaged to better determine the realism of the image from a global perspective. After the normalization operation of each convolutional layer, LeakyReLU is used as the activation function, and the α value is set to 0.2.

[0077] Figure 2A comparison table of the generation effects of this invention and various methods is disclosed. Each group, from top to bottom, represents the output under the labels of daytime, dusk, and nighttime. The lower right corner of each image in the NST output shows the selected style image. A suitable style image can produce good results, but in practical applications, it's difficult to guarantee that every input image has a suitable style image. CycleGAN's output has the colors of anime images, but the textures are too complex, and it cannot generate images with different temporal characteristics. MUNIT can generate style images with different temporal characteristics, but the images generated by MUNIT largely lose the original image structure. Adding a temporal conditional input to CartoonGAN and naming it cCartoonGAN results in output images with temporal characteristics, but the colors are dull and the textures are not smooth enough. This invention outputs colors with different temporal characteristics and smooth textures. The lines of fences and branches on the road become straight, and the background is filled with a uniform ambient color, a technique often used in game concept art. To make the evaluation more objective, the FID distance metric is used to evaluate the quality of the generated images, as shown in Table 1. FID (Fixed-Input Distance) is a metric used to evaluate the effectiveness of image processing. It uses a pre-trained ImageNet model to extract high-level features from an image, thereby calculating the distance between two image domains. When calculating FID for anime image domains with different temporal features, CycleGAN and CartoonGAN each use the same output domain, while other methods pair the outputs of specific time labels with the corresponding anime temporal feature domains for calculation. Therefore, as... Figure 2 As shown, this invention not only achieves the best results in the FID index of anime style, but also achieves the best results in the comparison of FID index in each time feature domain.

[0078] Example 2

[0079] To effectively save game developers' resources and time, and to configure the model on a web platform based on Tensorflow.js, allowing developers to focus on the creative aspects and improve the quality of their games, the program automatically generates animation images at three different time points based on the trained model. After the generation result viewing interface pops up, users can switch the time point through the options panel. Clicking "Save As" will bring up a file save dialog box, allowing users to save the current image to their computer's hard drive.

[0080] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0081] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to the above embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0082] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating an animation style game background based on a generative adversarial network, characterized in that, It comprises the following steps: S1, constructing a game background image, obtaining a data set in the game background image, each animation scene in the data set being composed of three images at different time nodes, and marking an attribute label i for each image; S2, constructing a color transfer network for generating a color segmentation map and a style reconstruction network for reconstructing an animation style map, using a pre-trained VGG network and a Gram matrix to construct a loss function, and optimizing the structure and style of the generated image; S3, using a generative adversarial network for animation style transfer in the model preparation stage using Python language, generating a line drawing and a color block drawing of the photo in the input stage using an edge detection algorithm and an image segmentation algorithm, and combining the line drawing and the color block drawing with the animation style picture for unsupervised training; S4, using an edge extraction algorithm to generate a line drawing corresponding to the training picture; S5, using a color segmentation algorithm to extract a color segmentation image of the picture in the data set; In step S1, the attribute label i takes values of 0, 1 and 2, when the attribute label takes a value of 0, it represents that the animation scene is in the daytime, when the attribute label takes a value of 1, it represents that the animation scene is in the dusk, and when the attribute label takes a value of 2, it represents that the animation scene is in the night. S6 uses the Python programming language to build network models for the color transfer and style reconstruction stages. and By collecting the dataset and Training, taking photos With time node labels Input generator Generation and Tags Corresponding color segmentation image Then and Edge detection images Input generator Generate target anime-style game background image .

2. The method of claim 1, wherein the method is based on a generative adversarial network. In step S1, the pictures in the data set with a proportion of 16:9 are processed as follows:

3. The method of claim 1, wherein the method is based on a generative adversarial network. The pictures in the data set with a proportion of 16:9 are cut from the middle to obtain two pictures with a proportion of 9:8, then the two pictures are adjusted to a size of 512x512, and finally the picture with a size of 512x512 is cut again into four pictures with a size of 256x256 using the half of the horizontal and vertical coordinates as the axis. The adversarial loss function is:

4. The method of claim 1, wherein the method is based on a generative adversarial network. The method for generating the line picture corresponding to the training picture using the edge extraction algorithm in step S4 is as follows: the total number of scenes in the game scene database is , represent the background picture under the time condition of the th scene, wherein the value range of t is , , the value of s is any one of 0, 1 and 2, for each scene, the picture of the daytime time node is selected as the base picture to generate the edge detection picture corresponding to the scene, if the training picture is , then the line picture corresponding thereto is .

5. The method of claim 4, wherein the method is based on a generative adversarial network. The color transfer network loss function is constructed, which consists of two parts: an adversarial loss function , a content loss function: ; wherein G1 refers to the generator of the color transfer stage, and D1 refers to the discriminator of the color transfer stage; the constructed adversarial loss function is: ； S is the Felzenszwalb image segmentation algorithm, log is the logarithmic function, E y~M refers to sampling a picture y from the data set M, E x~P refers to sampling a picture x from the data set P; the constructed content loss function is: ； where M is a set of anime pictures, is one of the anime pictures, is the corresponding time node label, is one of the pictures, is one of the pictures, represents the color segmentation algorithm, is a formula commonly used in video engineering to convert color to grayscale, and ||1 represents the calculation of the root mean square error, x~P refers to sampling a picture x from the dataset P, and VGG represents a pre-trained VGG network, , , are the red, green, and blue channels of the image, respectively, so that 。 6. The method of claim 1, wherein the method is based on a generative adversarial network. The loss function of the style reconstruction network is constructed, including an adversarial loss function , a content loss function , a style loss function , and a color segmentation loss function .

7. The method of claim 6, wherein the method is based on a generative adversarial network. The constructed content loss function is: ； wherein, is a Laplacian edge extraction operator, is the Felzenszwalb image segmentation algorithm G2 refers to the generator of the style reconstruction phase, D2 refers to the discriminator of the style reconstruction phase, x , refers to the picture generated in the color transfer phase, E y~M refers to sampling a picture y from the dataset M, E x~P refers to sampling a picture x from the dataset P; The constructed style loss function is: ； wherein VGG refers to a pre-trained VGG network, F gray refers to a formula for color conversion to grayscale, and ||1indicates the calculation of the root mean square error; The constructed color segmentation loss function is: ； Wherein, Gram represents a Gram matrix. ； The generators G1 and G2 comprise two parts of encoding and decoding; the encoding part comprises a full connection layer, a down-sampling layer and a plurality of residual blocks, each residual block is a jump connection, the decoding part comprises an up-sampling layer and a full connection layer, the size of each feature map is halved after each down-sampling of the generator, and the size of each feature map is doubled after each up-sampling.

8. The method of claim 1, wherein the method is based on a generative adversarial network. The platform comprises a browser and a server, the server generates a game background using the animation style game background generation method of any one of claims 1-8, a user directly calls a trained model stored in the background of the server to train the game background when using, and the browser is used to display the game background. 9.A platform for generating an animation style game background based on a generative adversarial network, characterized in that: ​

Images