A method and device for image refinement in animation based on adversarial neural network

Abstract

The present invention discloses a method and device for image refinement of animation based on adversarial neural network, belonging to the field of image generation technology, the method comprises: adding material classification information to the animation scene design drawing and converting it into a binary mask image, channel splicing the binary mask image and the animation scene design drawing to obtain a feature map; inputting the feature map into a generator network with an improved PSA module to generate an image to obtain a refined result map; synchronously performing alternating adversarial training on the main discriminator and the auxiliary discriminator with the generator network, and supervising the training in combination with a loss function, to obtain an optimized generator network as the final image refinement model; inputting a new animation scene design drawing into the image refinement model to generate an image to obtain a new refined result map. The present invention can improve the image generation capability of one-to-one precise control, effectively realize the improvement of image detail richness, and promote the development and application of image generation technology.

Description

Technical Field

[0001] The present invention belongs to the technical field of image generation, and in particular relates to a method and device for image refinement in animation production based on adversarial neural networks. Background Art

[0002] In the vast field of image generation technology, deep learning models are increasingly widely used. However, under precisely controlled input conditions, the generation results of traditional deep learning models are difficult to meet the required structural integrity and detail differentiation, and cannot achieve stable and ideal detail generation, resulting in the lack of quality and diversity of generated images. In the existing field of image generation technology, such as the production of style images of two-dimensional animation shots, it is necessary to use deep learning methods to improve the richness of image details. Existing technical solutions mainly focus on the following three aspects: matching training, non-matching training, and multi-domain fusion training.

[0003] (1) Matching training. In matching training, the source domain images and target images in the training set correspond one to one, providing a clear training target for the model. Pix2Pix is ​​a relatively common framework in this field. It is based on a deep convolutional generative adversarial network and uses convolution, batch normalization, and ReLU activation functions to design the network. In order to improve the quality and resolution of the generated images, the Pix2PixHD framework further divides the generator into two sub-networks, which are used to train images with lower and higher resolutions respectively, and introduces a multi-scale discriminator to distinguish between real images and generated images. In addition, the perceptual adversarial network (PAN) achieves a universal pixel-to-pixel image conversion by adding perceptual loss based on Pix2Pix. This type of method makes model training easier by constructing a more strict mapping relationship between input and output, but it is easy to fall into overfitting, resulting in a relatively simple generated result and a lack of diversity.

[0004] (2) Non-matching training. In the dataset of non-matching training, there is no structural correlation between the input and output images. It uses any image in the target domain as the label corresponding to the source domain, which reduces the difficulty of data acquisition. The more common frameworks include CycleGAN, DualGAN and DiscoGAN, all of which use cycle consistency structures to construct the mapping between the source domain and the target domain. Considering the weak effectiveness of cycle consistency constraints, Sem-GAN built a semantic consistency framework, and AttentionGAN based on the attention guidance mechanism made improvements on the generator. The above method makes the model more likely to generate diverse results, but there will be certain structural and texture distortions in the generated results.

[0005] (3) Multi-domain fusion training. There are images of multiple modalities in different fields, and there are cases of multi-domain mapping. StarGAN realizes the simultaneous mapping of datasets from different fields in a single network. In order to enable the generator to learn the ability to convert between multiple fields, it adds target field information to the generator. The discriminator needs to judge whether the image is real and to which field the image belongs, so as to ensure that the output image presents different effects depending on the target field. MNISE-GAN uses the front network of the discriminator as a style encoder, which enhances the ability to distinguish the styles of images of different modalities and simplifies the overall network structure. However, this type of method usually requires adding a series of external factors such as network input noise, which will reduce the stability of network training and has certain implementation difficulties.

[0006] In summary, the above-mentioned technical solutions have improved the image generation effect to a certain extent, but there are still some common problems: (1) The prompt information in the input is insufficient, so that the model lacks clear guidance when learning the mapping relationship; (2) The relevant network structure lacks internal adaptability, which limits the learning ability of the model; (3) The requirement for the amount of data is high. Under the condition of a small amount of training data, the generalization ability of the model is reduced, resulting in unsatisfactory generation effect; (4) The composition of the relevant loss function has great limitations, and its supervision method is not suitable for model training based on accurate generation. Therefore, it is necessary to provide a more accurate and high-quality image generation refinement method to promote the further development of image generation technology. Summary of the invention

[0007] In view of the above, the purpose of the present invention is to provide a method and device for animation image refinement based on adversarial neural networks, so as to solve the problem that the results generated by the deep learning model under precisely controlled input conditions are difficult to meet the required structural integrity and detail distinction and cannot achieve stable and ideal detail generation, realize one-to-one precise control of image generation capabilities, use deep learning methods to improve the richness of image details, promote the development of image generation technology, and provide new solutions for fields such as style image production of two-dimensional animation shots.

[0008] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0009] An embodiment of the present invention provides a method for image refinement of animation based on an adversarial neural network, comprising the following steps:

[0010] Add material classification information to the animation scene design diagram and convert it into a binary mask diagram, and perform channel splicing on the binary mask diagram and the animation scene design diagram to obtain a feature map;

[0011] The feature map is input into the generator network with the improved PSA module for image generation to obtain a refined result map, wherein in the improved PSA module, the high-frequency information obtained by global maximum pooling of the fine-grained feature map extracted based on the feature map and the low-frequency information obtained by global average pooling of the coarse-grained feature map extracted based on the feature map are fused to obtain the feature map after attention adjustment for the generation of the refined result map;

[0012] The main discriminator built based on the Unet network and the auxiliary discriminator built based on the pre-trained visual auxiliary network are synchronously trained with the generator network for alternating adversarial training. The training is supervised by the loss function, and the optimized generator network is obtained as the final image refinement model.

[0013] The new animation scene design graph is input into the image refinement model for image generation to obtain a new refinement result graph.

[0014] Preferably, the step of adding material classification information to the animation scene design diagram and converting it into a binary mask diagram, and performing channel splicing on the binary mask diagram and the animation scene design diagram to obtain a feature diagram comprises:

[0015] An animation scene design drawing with RGB three channels containing color and structure information is obtained and normalized. Material classification information is introduced on the basis of the normalized animation scene design drawing, that is, a material classification mask map with RGB three channels for distinguishing materials in different areas is introduced. The material classification mask map is converted into a binary mask map with the same number of material categories. All the binary mask maps are channel-joined and then channel-joined with the normalized animation scene design map to obtain a feature map as the input of the generator network.

[0016] Preferably, the step of inputting the feature map into a generator network with an improved PSA module to generate an image to obtain a refined result map comprises:

[0017] The feature map is input into the generator network. The feature map is first downsampled and convolved, and then input into the improved PSA module to extract the attention-adjusted feature map that fuses high-frequency information and low-frequency information. The attention-adjusted feature map is input into several RRDB modules and then subjected to convolution operations for feature extraction to obtain a feature extraction map. The feature extraction map and the attention-adjusted feature map are added together, and then upsampled and convolved to obtain the refined result map output by the generator network.

[0018] Preferably, when generating the feature map after attention adjustment in the improved PSA module, the feature map after downsampling and convolution operations is first extracted using convolution layers with different convolution kernel sizes to obtain fine-grained feature maps and coarse-grained feature maps, global maximum pooling is used for the fine-grained feature map to obtain the weight vector corresponding to the high-frequency information, and global average pooling is used for the coarse-grained feature map to obtain the weight vector corresponding to the low-frequency information. The weight vector corresponding to the high-frequency information and the weight vector corresponding to the low-frequency information are channel-concatenated to obtain a complete weight vector and normalized. The splicing result of the channel-concatenated fine-grained feature map and the coarse-grained feature map is dot-multiplied with the normalized complete weight vector to obtain the feature map after attention adjustment.

[0019] Preferably, the step of synchronously performing alternating adversarial training on a main discriminator constructed based on a Unet network and an auxiliary discriminator constructed based on a pre-trained visual auxiliary network with a generator network comprises:

[0020] The main discriminator constructed based on the Unet network and the auxiliary discriminator constructed based on the pre-trained visual auxiliary network together constitute a set of discriminators opposite to the generator network. The discriminator is used to judge the refined result image output by the generator network as false and the real hand-drawn target image as true. The parameters of the discriminator and generator network are updated through alternating adversarial training. Among them, for the auxiliary discriminator, the parameters of each layer belonging to the feature extraction part are fixed, and only the network parameters of the shallow classifier are updated.

[0021] Preferably, the loss function Includes content loss between the refined result map and the true hand-drawn target map , style loss , Similarity loss , and the adversarial loss of the generator network , the formula is as follows:

[0022] ,

[0023] in, Express expectations, represents the input feature map, represents the distribution of the input domain, and denote the main discriminator and the auxiliary discriminator respectively, represents the generator network.

[0024] Preferably, the content loss The calculation includes:

[0025] The refined result image and the real hand-drawn target image are respectively input into the VGG19 network to extract two high-level content representations. The distance value between the two high-level content representations is calculated as the consistency loss value between the refined result image and the real hand-drawn target image in the high-level content representation, and is recorded as the content loss .

[0026] Preferably, the style loss The calculation includes:

[0027] Based on the high-level content representation extracted by the pre-trained VGG19 network, its corresponding high-level style representation is calculated, and the distance value between the two high-level style representations is calculated as the consistency loss value between the refined result image and the real hand-painted target image in the high-level style representation, which is recorded as the style loss .

[0028] Preferably, the similarity loss The calculation includes:

[0029] Based on the pre-trained similarity evaluation model, the consistency loss value between the refined result image and the real hand-drawn target image in terms of similarity is calculated and recorded as the similarity loss .

[0030] To achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a device for image thinning of animation production based on adversarial neural network, which is implemented by the above-mentioned method for image thinning of animation production based on adversarial neural network, and includes: a feature map construction module, an image thinning module, an adversarial training module, and a model application module;

[0031] The feature map construction module is used to add material classification information to the animation scene design map and convert it into a binary mask map, and perform channel splicing on the binary mask map and the animation scene design map to obtain a feature map;

[0032] The image refinement module is used to input the feature map into a generator network with an improved PSA module to generate an image and obtain a refined result map, wherein in the improved PSA module, the high-frequency information obtained by global maximum pooling of a fine-grained feature map extracted based on the feature map and the low-frequency information obtained by global average pooling of a coarse-grained feature map extracted based on the feature map are fused to obtain an attention-adjusted feature map for generating a refined result map;

[0033] The adversarial training module is used to perform alternating adversarial training on the main discriminator constructed based on the Unet network and the auxiliary discriminator constructed based on the pre-trained visual auxiliary network synchronously with the generator network, and supervise the training in combination with the loss function, so as to obtain the optimized generator network as the final image refinement model;

[0034] The model application module is used to input a new animation scene design diagram into the image refinement model to perform image generation to obtain a new refinement result diagram.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) To address the problem of insufficient prompt information in the input, the present invention adds a material classification mask as additional information on the basis of the original input animation scene design diagram, thereby strengthening the guidance for network training and helping to improve recognition accuracy and learning efficiency.

[0037] (2) In view of the structural defects of the existing generator network, the present invention introduces the PSA module into the generator network and makes further improvements to the module. The high-frequency information and low-frequency information of the fine-grained feature map and the coarse-grained feature map are obtained and fused through global maximum pooling and global average pooling respectively. The feature map after attention adjustment is used to generate the refined result map, so that the generator network has a better generation effect.

[0038] (3) To address the problem of limited training data leading to poor training results, the present invention introduces a pre-trained visual auxiliary network as an auxiliary discriminator, which can more effectively extract visual features from small-scale data based on certain prior knowledge, and combines the main discriminator constructed by the Unet network as a group of discriminators for adversarial training with the generator network, thereby reducing overfitting and providing a more stable training process under the condition of a small amount of training data, thereby enhancing the training effect.

[0039] (4) To address the problem that traditional loss functions cannot effectively supervise accurate generative models, the present invention proposes a self-balancing loss function construction method that integrates loss value calculation indicators from different angles, so that the model can continuously tend towards a more reasonable supervision intensity during the training process, achieving a balance between structural integrity and detail richness in the generated results. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 It is a flowchart of a method for image refinement in animation production based on an adversarial neural network provided by an embodiment of the present invention;

[0042] Figure 2 It is a schematic diagram of a process of adding material classification information to construct a feature map provided by an embodiment of the present invention;

[0043] Figure 3 It is a schematic diagram of the structure and flow of a generator network provided by an embodiment of the present invention;

[0044] Figure 4 It is a schematic diagram of the structure and flow of an improved PSA module provided in an embodiment of the present invention;

[0045] Figure 5 It is a schematic diagram of the overall process of adversarial training provided by an embodiment of the present invention;

[0046] Figure 6 It is a structural schematic diagram of a device for animation image refinement based on an adversarial neural network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0048] The inventive concept of the present invention is: in response to the problems existing in the image generation method based on deep learning in the prior art, such as insufficient prompt information in the input, lack of internal adaptability of the network structure, limited training data and limitations of the loss function, the embodiment of the present invention provides a method and device for animation image refinement based on an adversarial neural network, by adding a mask map of material classification information as additional information, and introducing an improved PSA module in the generator network, while the generator is alternately trained with the main discriminator and the auxiliary discriminator respectively, and the loss value calculation indicators at different angles are integrated based on a self-balancing loss function construction method, so as to train the optimized generator network as the final image refinement model, thereby effectively improving the image generation capability and image generation effect of the model, and achieving the improvement of the richness of image details while ensuring the structural integrity.

[0049] Figure 1 FIG. 1 is a flow chart of a method for image refinement in animation based on an adversarial neural network provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a method for image refinement of animation based on adversarial neural network, comprising the following steps:

[0050] S1, adding material classification information to the animation scene design diagram and converting it into a binary mask diagram, and channel-joining the binary mask diagram and the animation scene design diagram to obtain a feature map.

[0051] In the embodiment, Figure 2As shown, an RGB three-channel animation scene design diagram containing color and structure information is obtained and normalized. In order to enhance the guidance of the input information for the accurate generation of the model, material classification information is introduced on the basis of the normalized animation scene design diagram that only includes color and structure information, that is, a material classification mask of RGB three channels is added to distinguish materials in different regions. In the embodiment, a total of 14 types of materials are added. Considering that the network is more sensitive to binary input, the colored material classification mask is converted into a binary mask with the same number of material categories, and a total of 14 binary mask diagrams are generated. All binary mask diagrams are channel-joined and then channel-joined with the normalized animation scene design diagram to form a feature map with more complete prompt information and input it into the generator network, so that the generator network can more effectively learn the mapping relationship between input and output during the training process.

[0052] S2, the feature map is input into the generator network with the improved PSA module for image generation to obtain a refined result map.

[0053] In the embodiment, in order to realize the training of the generator network under the condition of limited video memory, a pixel re-arrangement layer (pixel_unshuffle) is added to the generator network, which performs downsampling operation on the input feature map, and the downsampling coefficient is set to 4. While partially expanding the number of channels (C), the length (H) and width (W) of the feature map are reduced to 1 / 4 of the original size. The above operation effectively reduces the amount of video memory required by the model during the training process while retaining the feature map information. Considering that the image size output by the generator network needs to be the same as the length and width of the input image, two upsampling layers are added before the output layer of the generator network, and the upsampling coefficients are all set to 2, and the interpolation mode is set to nearest, and the convolution layer is combined to achieve the final output. In addition, to ensure that a larger batch_size can be maintained under specific video memory limitations to achieve better training results, before training, the images of different sizes in the training data and their corresponding mask images are divided into 512*512 image blocks with a step size of 256. During the training process, a 256*256 image block is randomly intercepted from each of the above image blocks during one iteration. The above operation aims to save video memory while learning different data distribution relationships in a balanced manner. At this time, batch_size is set to 32, that is, at each iteration, the data tensor of a batch learned by the generator network is 32*256*256.

[0054] In order to make the generator network better balance the generation intensity differences between different regions during the training process, the Pyramid Split Attention (PSA) module is introduced and improved. In the original PSA module, global average pooling is used for feature induction for feature maps of different granularities. The angle is single and there is a lack of targeted processing for feature maps of different granularities. Therefore, the embodiment proposes to use global average pooling for induction of coarse-grained feature maps to obtain more effective low-frequency information, and global maximum pooling for induction of fine-grained feature maps to obtain more effective high-frequency information. Finally, the above two are combined to form the basis for weight allocation of feature maps, so that the attention mechanism can play its role more comprehensively and accurately.

[0055] The overall structure and processing flow of the generator network are as follows Figure 3 As shown, the feature map In the input generator network, the feature map is first downsampled by the pixel rearrangement layer and then extracted by the convolution layer. The feature map is then input into the improved PSA module. The attention-adjusted feature map that fuses high-frequency information and low-frequency information is extracted in the improved PSA module. The attention-adjusted feature map is input into a 23-layer residual nested dense module (Residual-in-Residual Dense Block, RRDB) and then extracted by the convolution layer to obtain a feature extraction map. The feature extraction map and the attention-adjusted feature map are added together, and then the upsampling operation of the upsampling layer and the feature extraction of the convolution layer are performed. Finally, after three convolution layers, the refined result map output by the generator network is obtained.

[0056] The overall structure and processing flow of the improved PSA module are as follows: Figure 4 As shown in the figure, first input the feature map after downsampling and convolution operation, and extract the features through the convolution layers with convolution kernel sizes of 3*3 and 5*5 to obtain two fine-grained feature maps, and extract the features through the convolution layers with convolution kernel sizes of 7*7 and 9*9 to obtain two coarse-grained feature maps. The two fine-grained feature maps are passed through the global maximum pooling attention module to obtain two weight vectors corresponding to high-frequency information, and the coarse-grained feature map is passed through the global average pooling attention module to obtain two weight vectors corresponding to low-frequency information. The weight vectors corresponding to the two high-frequency information and the weight vectors corresponding to the two low-frequency information are channel-joined to obtain a complete weight vector, which is then normalized by softmax. The splicing result of the channel-joining of the two fine-grained feature maps and the two coarse-grained feature maps is dot-multiplied with the normalized complete weight vector to obtain the feature map after attention adjustment.

[0057] The internal calculation process of the improved PSA module is as follows:

[0058] ,

[0059] The calculations of each item inside are as follows:

[0060] ,

[0061] ,

[0062] ,

[0063] ,

[0064] in, represents the feature map after downsampling and convolution operation input to the improved PSA module, Indicates the improved PSA module, Indicates channel splicing, represents dot product, , , and They represent convolutional layers with kernel sizes of 3*3, 5*5, 7*7, and 9*9, respectively. represents the global average pooling attention module, represents the global maximum pooling attention module, represents the coarse-grained feature map or fine-grained feature map input to the above two modules, represents the ReLU activation function, represents the sigmoid activation function, , , and Represents the parameter matrix of each fully connected layer in the attention module. express The vector obtained by global average pooling, express The vector obtained by global maximum pooling, Represents the first input coarse-grained feature map or fine-grained feature map The characteristics of the channels, Indicates that Calculate the result of global average pooling, that is, vector Middle The value of the bit, Indicates that Calculate the result of the global maximum pooling, that is, the vector Middle The value of the bit, and Respectively The length and width of and Pixel indices for length and width respectively.

[0065] S3, the main discriminator built based on the Unet network and the auxiliary discriminator built based on the pre-trained visual auxiliary network are synchronously trained with the generator network for alternating adversarial training, and the training is supervised by combining the loss function to obtain the optimized generator network as the final image refinement model.

[0066] In the embodiment, in order to solve the problem of poor training effect caused by small-scale data sets, a pre-trained visual aided network (Vision-aidedNet) is introduced as an auxiliary discriminator. During training, the parameters of each layer of the auxiliary discriminator belonging to the feature extraction part are fixed, and only the network parameters of the shallow classifier are updated, so that the auxiliary discriminator can more effectively extract visual features from small-scale data based on certain prior knowledge. It and the main discriminator built based on the Unet network together constitute a group of discriminators opposite to the generator, thereby reducing overfitting and providing a more stable training process. During training, such as Figure 5 As shown in the figure, the generator and the discriminator are trained alternately against each other, where the generator is responsible for generating images that are as realistic as possible, that is, close to the hand-drawn quality, so that the discriminator recognizes them as real. The discriminator is responsible for distinguishing the refined result image generated by the generator network from the real hand-drawn target image, that is, recognizing the real hand-drawn target image as real and the generated refined result image as fake. The training objectives of the above generator network and the discriminator network are opposite, presenting an alternating adversarial training mode. In this process, the network is fed back and updated through the discrimination results of the main discriminator and the auxiliary discriminator, so that the performance of both the generator and the discriminator networks are continuously improved. After the training is completed, the generator network is left as the final image refinement model.

[0067] The objective function of the complete generative adversarial network is:

[0068] ,

[0069] in, Express expectations, Represents a real hand-drawn target map, represents the distribution of the target domain, and denote the main discriminator and the auxiliary discriminator respectively, represents the input feature map, represents the distribution of the input domain, represents the generator network.

[0070] In the embodiment, in order to make the generator network generate results with both structural integrity and rich details, the supervision of training needs to maintain a reasonable intensity. Therefore, a self-balancing loss function construction method is proposed, that is, the loss value calculation indicators from different angles are integrated, so that it automatically tends to a reasonable supervision intensity during the training process. The corresponding real hand-drawn target map There is relative consistency between them in deep representation. The pre-trained VGG19 network is used to calculate the deep feature map, and the output feature map of the 4th convolution layer (conv_5_4) in the 5th convolution module is used as the high-level content representation. Specifically, the refined result map and the real hand-drawn target map are respectively input into the pre-trained VGG19 network, and the output feature maps of the conv_5_4 layer are respectively taken as two high-level content representations. The distance value between the two high-level content representations is calculated as the consistency loss value between the generated result and the target result in the high-level content representation, and is used as one of the indicators of the supervised generator, recorded as content loss At the same time, based on the above high-level content representation, the corresponding high-level style representation is calculated as another indicator of the supervised generator. The distance between the two high-level style representations is used as the consistency loss value between the refined result image and the real hand-painted target image in the high-level style representation, and is recorded as the style loss , in order to balance the overfitting that may be caused by strong consistency in content representation. In addition, the pre-trained similarity evaluation model topiq_fr_pipal is introduced as a comprehensive trade-off indicator to calculate the consistency loss value of the similarity between the refined result image and the real hand-drawn target image, and it is recorded as the similarity loss , to enhance the stability of the supervision process. To further enhance the freedom of the generator, the style loss is multiplied by the similarity loss to increase the possibility of the generator changing in different directions during training. Combined with the aforementioned adversarial loss involving the generator network, the overall loss function of the generator is constructed as follows:

[0071] ,

[0072] The calculation formulas for each loss value are as follows:

[0073] ,

[0074] ,

[0075] ,

[0076] ,

[0077] in, represents the 4th convolutional layer in the 5th convolutional module in the pre-trained VGG19 network, Represents the parameters of the pre-trained similarity evaluation model topiq_fr_pipal, Represents the calculation of the style representation of the feature map, where H, W, and C represent the length, width, and number of channels of the input feature map, respectively. , and are the indices of length, width and number of channels respectively, Represents high-level content representation. The L1 norm is used in the function The distance loss is measured to suppress the adverse effects of deviating data on training, thereby enhancing the robustness of the training process.

[0078] By combining the loss function to supervise the training, the optimized generator network is obtained as the final image refinement model.

[0079] S4, inputting the new animation scene design graph into the image refinement model for image generation to obtain a new refinement result graph.

[0080] In specific applications, new animation scene design drawings that need to be refined are input into the image refinement model, and new refinement result maps that meet the requirements of structural integrity and detail distinction are output for use in style image production of two-dimensional animation shots, thereby improving the richness of image details.

[0081] Based on the method for animation image refinement based on adversarial neural network provided in the above embodiment, the superiority of the proposed method is verified through ablation experiments, and the relevant results are shown in Table 1 below:

[0082] Table 1 Comparison of experimental results

[0083]

[0084] In Table 1, the objective indicators of the model generation results of the initial method and the model generation results after the introduction of relevant optimization strategies are shown respectively, where RRDBNet is the initial generator network, Unet is the discriminator network, mask represents the material mask information added to the input, RRDBNet (w / PSA) represents the generator network after the introduction of the initial PSA module, RRDBNet (w / improved-PSA) represents the generator network after the introduction of the improved PSA module, Vision-AidedNet represents the introduced visual aid network, and self-balanced-loss represents the self-balanced loss function. It can be observed that the introduction of various optimization strategies has a significant improvement on the generation results of the model, which is manifested in the gradual improvement of the peak signal-to-noise ratio (PSNR) and the gradual reduction of the learning perceptual image block similarity (LPIPS). Finally, the overall solution proposed has improved the PSNR index by 3.46 and reduced the LPIPS index by 0.15 compared with the initial method, which reflects the superiority of the present invention.

[0085] Based on the same inventive concept, Figure 6 As shown, an embodiment of the present invention also provides a device 600 for animation image refinement based on an adversarial neural network, including: a feature map construction module 610, an image refinement module 620, an adversarial training module 630, and a model application module 640.

[0086] The feature map construction module 610 is used to add material classification information to the animation scene design map and convert it into a binary mask map, and perform channel splicing on the binary mask map and the animation scene design map to obtain a feature map.

[0087] The image refinement module 620 is used to input the feature map into the generator network with the improved PSA module to generate an image to obtain a refined result map, wherein, in the improved PSA module, the high-frequency information obtained by global maximum pooling of the fine-grained feature map extracted based on the feature map and the low-frequency information obtained by global average pooling of the coarse-grained feature map extracted based on the feature map are fused to obtain the feature map after attention adjustment for generating the refined result map.

[0088] The adversarial training module 630 is used to perform alternating adversarial training on the main discriminator constructed based on the Unet network and the auxiliary discriminator constructed based on the pre-trained visual auxiliary network synchronously with the generator network, and supervise the training in combination with the loss function to obtain the optimized generator network as the final image refinement model.

[0089] The model application module 640 is used to input the new animation scene design diagram into the image refinement model to perform image generation to obtain a new refinement result diagram.

[0090] It should be noted that the device for animation image refinement based on adversarial neural network provided in the above embodiment belongs to the same inventive concept as the method for animation image refinement based on adversarial neural network. The specific implementation process is detailed in the embodiment of the method for animation image refinement based on adversarial neural network, which will not be repeated here.

[0091] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for image refinement of animation based on adversarial neural network, characterized in that: The following steps are involved: Add material classification information to the animation scene design diagram and convert it into a binary mask diagram, and perform channel splicing on the binary mask diagram and the animation scene design diagram to obtain a feature map; The feature map is input into the generator network with the improved PSA module for image generation to obtain a refined result map, wherein in the improved PSA module, the high-frequency information obtained by global maximum pooling of the fine-grained feature map extracted based on the feature map and the low-frequency information obtained by global average pooling of the coarse-grained feature map extracted based on the feature map are fused to obtain the feature map after attention adjustment for the generation of the refined result map; The main discriminator built based on the Unet network and the auxiliary discriminator built based on the pre-trained visual auxiliary network are synchronously trained with the generator network for alternating adversarial training. The training is supervised by the loss function, and the optimized generator network is obtained as the final image refinement model. The new animation scene design graph is input into the image refinement model for image generation to obtain a new refinement result graph.

2. The method for image refinement of animation based on adversarial neural network according to claim 1, characterized in that: The method of adding material classification information to the animation scene design diagram and converting it into a binary mask diagram, and performing channel splicing on the binary mask diagram and the animation scene design diagram to obtain a feature diagram includes: An animation scene design drawing with RGB three channels containing color and structure information is obtained and normalized. Material classification information is introduced on the basis of the normalized animation scene design drawing, that is, a material classification mask map with RGB three channels for distinguishing materials in different areas is introduced. The material classification mask map is converted into a binary mask map with the same number of material categories. All the binary mask maps are channel-joined and then channel-joined with the normalized animation scene design map to obtain a feature map as the input of the generator network.

3. The method for animation image refinement based on adversarial neural network according to claim 1, characterized in that: The feature map is input into the generator network with the improved PSA module to generate an image to obtain a refined result map, including: The feature map is input into the generator network. The feature map is first downsampled and convolved, and then input into the improved PSA module to extract the attention-adjusted feature map that fuses high-frequency information and low-frequency information. The attention-adjusted feature map is input into several RRDB modules and then subjected to convolution operations for feature extraction to obtain a feature extraction map. The feature extraction map and the attention-adjusted feature map are added together, and then upsampled and convolved to obtain the refined result map output by the generator network.

4. The method for animation image refinement based on adversarial neural network according to claim 1 or 3, characterized in that: When generating the feature map after attention adjustment in the improved PSA module, the feature map after downsampling and convolution operations is first extracted using convolution layers with different convolution kernel sizes to obtain fine-grained feature maps and coarse-grained feature maps, and the global maximum pooling is used for the fine-grained feature map to obtain the weight vector corresponding to the high-frequency information, and the global average pooling is used for the coarse-grained feature map to obtain the weight vector corresponding to the low-frequency information. The weight vector corresponding to the high-frequency information and the weight vector corresponding to the low-frequency information are channel-concatenated to obtain the complete weight vector and normalized. The splicing result of the fine-grained feature map and the coarse-grained feature map is dot-multiplied with the normalized complete weight vector to obtain the feature map after attention adjustment.

5. The method for animation image refinement based on adversarial neural network according to claim 1, characterized in that: The method of synchronously performing alternating adversarial training on a main discriminator constructed based on a Unet network and an auxiliary discriminator constructed based on a pre-trained visual auxiliary network with a generator network includes: The main discriminator constructed based on the Unet network and the auxiliary discriminator constructed based on the pre-trained visual auxiliary network together constitute a set of discriminators opposite to the generator network. The discriminator is used to judge the refined result image output by the generator network as false and the real hand-drawn target image as true. The parameters of the discriminator and generator network are updated through alternating adversarial training. Among them, for the auxiliary discriminator, the parameters of each layer belonging to the feature extraction part are fixed, and only the network parameters of the shallow classifier are updated.

6. The method for animation image refinement based on adversarial neural network according to claim 1, characterized in that: Loss Function Includes content loss between the refined result map and the true hand-drawn target map , style loss , Similarity loss , and the adversarial loss of the generator network , the formula is as follows: , in, Express expectations, represents the input feature map, represents the distribution of the input domain, and denote the main discriminator and the auxiliary discriminator respectively, represents the generator network.

7. The method for image refinement of animation based on adversarial neural network according to claim 6, characterized in that: Content loss The calculation includes: The refined result image and the real hand-drawn target image are respectively input into the VGG19 network to extract two high-level content representations. The distance value between the two high-level content representations is calculated as the consistency loss value between the refined result image and the real hand-drawn target image in the high-level content representation, and is recorded as the content loss .

8. The method for animation image refinement based on adversarial neural network according to claim 7, characterized in that: Style Loss The calculation includes: Based on the high-level content representation extracted by the pre-trained VGG19 network, its corresponding high-level style representation is calculated. The distance between the two high-level style representations is calculated as the consistency loss value between the refined result image and the real hand-painted target image in the high-level style representation, and is recorded as the style loss .

9. The method for animation image refinement based on adversarial neural network according to claim 6, characterized in that: Similarity loss The calculation includes: Based on the pre-trained similarity evaluation model, the consistency loss value between the refined result image and the real hand-drawn target image in terms of similarity is calculated and recorded as the similarity loss .

10. A device for thinning an animation image based on an adversarial neural network, implemented by the method for thinning an animation image based on an adversarial neural network according to any one of claims 1 to 9, characterized in that: include: Feature map construction module, image refinement module, adversarial training module, and model application module; The feature map construction module is used to add material classification information to the animation scene design map and convert it into a binary mask map, and perform channel splicing on the binary mask map and the animation scene design map to obtain a feature map; The image refinement module is used to input the feature map into a generator network with an improved PSA module to generate an image and obtain a refined result map, wherein in the improved PSA module, the high-frequency information obtained by global maximum pooling of a fine-grained feature map extracted based on the feature map and the low-frequency information obtained by global average pooling of a coarse-grained feature map extracted based on the feature map are fused to obtain an attention-adjusted feature map for generating a refined result map; The adversarial training module is used to perform alternating adversarial training on the main discriminator constructed based on the Unet network and the auxiliary discriminator constructed based on the pre-trained visual auxiliary network synchronously with the generator network, and supervise the training in combination with the loss function, so as to obtain the optimized generator network as the final image refinement model; The model application module is used to input a new animation scene design diagram into the image refinement model to perform image generation to obtain a new refinement result diagram.

Images