Image generation method and related device

Abstract

The present application discloses an image generation method and related devices, the image generation method comprising: obtaining an original image in a first domain; using a first image generation network to convert the original image into an initial image in a second domain; wherein the first image generation network is obtained by replacing part of the convolutional layers corresponding to the image generator network in the first domain with part of the convolutional layers of the image generator network in the second domain, and the image generator network in the second domain is obtained by transfer learning the image generator network in the first domain using a data set in the second domain; the initial image is repaired by the second image generation network to obtain an image corresponding to the original image in the second domain; wherein the second image generation network is obtained by multiple iterative training of the image generator network in the second domain. Through the above method, the image conversion from the image in the first domain to the image in the second domain is realized, and the generation quality of the image in the second domain is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of deep learning network technology, and in particular to an image generation method and related devices. Background Art

[0002] In recent years, various virtual images have sprung up like mushrooms after a rain, but many virtual images only focus on the image and ignore the long-term personality and content shaping. Except for a few top UP hosts, most virtual images lack tailor-made content.

[0003] According to the survey, only 20% of anchors are willing to turn on the camera, and the remaining 80% of anchors use static pictures as their virtual images to interact with the audience. The anchor uploads the avatar photo and generates a two-dimensional virtual image based on the anchor's avatar, thus forming a stylized image of the anchor with thousands of faces. At the same time, combined with the special effects and voice commands of the smart assistant, it can become an animation image output. The stylized image constitutes a virtual expression of the anchor, which can trigger the anchor's emotional projection and establish a connection between the anchor and the personalized image of thousands of faces, and the personalized image and the audience, while the content itself is still given by the anchor. Furthermore, when this link is optimized from the cloud to the mobile phone, the generation of the thousands of faces image and the action library are all generated offline and automatically, and played in the form of sequence frames on the mobile phone. At this time, the computing cost on the cloud is close to 0, and it can be fully covered on high and low terminals. This is also a reflection of all live broadcast platforms helping users to create content in a technology-driven way, while reducing the threshold and cost of art creation and improving the fun of content.

[0004] From the perspective of artificial intelligence algorithms, the difficulty in generating corresponding two-dimensional virtual images from real face pictures is to construct paired data sets of real-life domains and two-dimensional domains. Based on these paired data sets, a generative network from the face domain to the two-dimensional domain is trained by using supervised learning methods in image translation. To build such a data set, the most common method is for professional artists to draw corresponding two-dimensional virtual images for each face picture. However, this method is very time-consuming and very expensive. Moreover, the "paired data" before reproducing the European and American two-dimensional style from the Japanese two-dimensional style cannot be reused. Summary of the invention

[0005] The main technical problem solved by this application is to provide an image generation method and related devices to provide anchors with two-dimensional virtual images of different styles, reducing labor costs and time-consuming problems.

[0006] In order to solve the above technical problems, the present application provides an image generation method, wherein the image generation method includes: acquiring an original image in a first domain; using a first image generation network to convert the original image into an initial image in a second domain; wherein the first image generation network is obtained by replacing some convolutional layers corresponding to the image generator network in the first domain with some convolutional layers of the image generator network in the second domain, and the image generator network in the second domain is obtained by performing transfer learning on the image generator network in the first domain using a data set in the second domain; repairing the initial image through the second image generation network to obtain an image corresponding to the original image in the second domain; wherein the second image generation network is obtained by multiple iterative training of the image generator network in the second domain.

[0007] Among them, the step of converting the original image into the initial image of the second domain using the first image generation network includes: using a deep learning algorithm to detect the face of the original image; in response to the detected face being a side face image, aligning the side face into a front face image; and converting the front face image into the initial image of the second domain.

[0008] Among them, before the step of obtaining the original image of the first field, it includes: obtaining an image generator network of the first field and a data set of the second field; adjusting the data set through a super-resolution algorithm to obtain a data set of a set resolution; using the image generator network of the first field as the initial network, and using the data set of the set resolution to perform migration iterative learning on the initial network to obtain an image generator network of the second field.

[0009] Among them, the data set includes a second field image; before the step of adjusting the data set by a super-resolution algorithm to obtain a data set of a set resolution, it also includes: using a deep learning algorithm to perform face detection on the second field image; in response to the detected face being a side face image, aligning the side face to a front face image; and screening out the data set of the second field that contains face images.

[0010] Among them, the image generator of the first field and the image generator of the second field both include a convolution layer for collecting first facial features and a convolution layer for collecting second facial features; wherein the first facial features include at least one of face shape, expression and hairstyle, and the second facial features include at least color; after the step of using the image generator network of the first field as the initial network, performing migration iterative learning on the initial network using a data set of a set resolution to obtain the image generator network of the second field, it also includes: replacing the convolution layer of the second facial feature of the second field with the convolution layer of the second facial feature of the first field to obtain the first image generation network.

[0011] Among them, the number of iterations of the image generator network in the second field is less than the first set number and greater than the second set number.

[0012] Among them, the number of iterative training times of the second image generation network is greater than the number of iterations of the image generator network of the second domain.

[0013] Among them, the first realm is the real-life realm, and the second realm is the two-dimensional realm.

[0014] Among them, after the step of repairing the initial image through the second image generation network to obtain the image corresponding to the second field of the original image, it also includes: using style fusion technology and truncation techniques to modify the style attributes of the image.

[0015] The image generator network includes collecting a 512-dimensional z vector and mapping the z vector to an 18×512-dimensional latent vector space using a fully connected layer network; the 1-7 layers of latent vectors include collecting at least one feature of face shape, posture, hairstyle and expression, and the 8-18 layers of latent vectors include collecting at least one feature of background color, hair color and a second domain style; the step of modifying the style attributes of an image using style fusion technology and truncation techniques includes: modifying the feature latent vector of the image generator network using style fusion technology and truncation techniques to modify the style attributes of the image.

[0016] In order to solve the above technical problems, the present application also provides an intelligent terminal, which includes a processor and a memory coupled to each other, the memory is used to store program instructions, and the processor is used to execute the program instructions stored in the memory to implement the image generation method of any of the above embodiments.

[0017] In order to solve the above technical problems, the present application also provides a computer-readable storage medium, which includes a processor and a memory, the memory stores computer program instructions, and the processor is used to execute the program instructions to generate an image according to any of the above embodiments.

[0018] The beneficial effects of the present application are: using the first image generation network to convert the original image into an initial image of the second domain, and repairing the initial image through the second image generation network to obtain the image corresponding to the original image in the second domain, helping the anchor to automatically generate a virtual second domain image, reducing the threshold and cost of art creation, and improving the interest of live broadcast content. Based on the second-order optimization network, the generation quality of the second domain image is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of an implementation method of the image generation method of the present application;

[0020] Figure 2This is a flow chart of an implementation of a first image generation network generation method of the present application;

[0021] Figure 3 A schematic diagram of the structure of an implementation scheme of an image generator network in the second field of this application;

[0022] Figure 4 This is a structural diagram of an implementation method of convolutional layer exchange of the present application;

[0023] Figure 5 This is a structural diagram of an application scenario of the image generation network of this application;

[0024] Figure 6 This is a flow chart of another embodiment of the image generation method of the present application;

[0025] Figure 7 A schematic diagram of a flow chart for generating a second domain image generator network implementation method for this application;

[0026] Figure 8 This is a schematic diagram of a framework of an implementation method of a multi-style attribute control method of the present application;

[0027] Fig. 9 This is a schematic diagram of the structure of an implementation scheme of a smart terminal of the present application;

[0028] Fig.10 This is a schematic diagram of the structure of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] It should be noted that image translation is a technology for converting images from one data domain to another in computer vision. Generating a single image or multiple images from a picture based on the generative network can be divided into one-to-one image translation and one-to-many image translation problems. For the one-to-one image translation problem, the dataset used is paired data or unpaired data. The methodology is divided into supervised training methods and unsupervised learning methods, and different methods have their own advantages and disadvantages.

[0031] The supervised training method requires a paired dataset between the two domains. The advantage of this method is that the training and generation results are relatively stable, and the generation effect is also the best. However, the construction of paired datasets is relatively difficult. Different additional operations and steps are required according to different problems. Taking the image translation from the real domain to the two-dimensional domain as an example, in order to obtain paired data, artists need to manually draw each two-dimensional picture. Representative methods of supervised training include pix2pix (pixel-to-pixel), pix2pixHD (pixel-to-pixel high definition) and pSp (pixel-to-style-to-pixel) methods. Unsupervised training methods can train generative networks based on unpaired data. The advantage of this method is that it is relatively simple to obtain large batches of data, and only needs to collect data from two domains independently. But the disadvantage is that the generation results are unstable, and the generation results are poor compared to supervised training methods. Representative methods of unsupervised training include CycleGAN (Cycle-consistent GAN) and DiscoGAN (Discovery GAN). One-to-many image translation is based on multimodal generative network methods. Different methods use different definitions of multimodality, which can be randomly sampled features from a Gaussian distribution or a feature vector representation extracted from another image. Both supervised training and unsupervised training can define multimodality, the only difference is the use of paired and unpaired data.

[0032] The convolutional layer swapping technique is based on StyleGAN (a generative adversarial network). It first trains a generative network that randomly generates data in domain A, and then uses the weight of this network as the initialization to perform transfer learning on data in domain B, and trains the generative network in domain B. Then, the two generative networks in domain A and domain B are exchanged with each other to construct a new A-to-B generative network. Unlike image translation technology, the convolutional layer swapping technique is designed to achieve image conversion from one domain (domain A) to another domain (domain B). The convolutional layer swapping technique is an unsupervised training method, so paired data is not required. At the same time, because the transfer learning method is used, the data volume requirement is not high. This is an advantage that the image translation method does not have. At the same time, combined with feature fusion and latent vector attribute control, the newly constructed A-to-B generative network can also support one-to-many image generation and can achieve attribute control of generated images. However, because the A-to-B generative network is similar to the weight grafting between two networks, the quality of the generated image is poor, and the face and background are prone to contain visible defects.

[0033] This application proposes a two-dimensional image generation method based on second-order optimization on the convolutional layer exchange technology, which improves the image generation quality and solves the problem of defects in the image.

[0034] This application proposes an image generation method, see Figure 1 , Figure 1 FIG. 1 is a flow chart of an embodiment of the image generation method of the present application. Figure 1 As shown, the image generation method includes:

[0035] Step S11: Acquire the original image of the first area.

[0036] The first domain may be a real person domain or other domains.

[0037] In a specific implementation, the original image in the first area is a real-life image of the anchor. The acquisition method includes the anchor uploading the original image including the anchor's head portrait or taking a screenshot of the live broadcast to obtain the original image including the anchor's head portrait, which is not limited here.

[0038] Step S12: Utilize the first image generation network to convert the original image into an initial image of the second domain.

[0039] The second field may be a two-dimensional field or other fields.

[0040] In this embodiment, the first domain is a real person domain, and the second domain is a two-dimensional domain. The real person domain is a real person image composed of real person portraits, and the two-dimensional domain is a cartoon image composed of non-real person portraits.

[0041] In this embodiment, the first image generation network is obtained by replacing some convolutional layers corresponding to the image generator network in the first field with some convolutional layers of the image generator network in the second field. The image generator network in the second field is obtained by performing transfer learning on the image generator network in the first field using the data set in the second field.

[0042] The specific generation steps of the first image generation network are as follows: Please refer to Figure 2 , Figure 2 This is a flow chart of an implementation of the first image generation network generation method of the present application. It includes:

[0043] Step S21: Obtain the image generator network of the first domain.

[0044] The image generator network of the first domain includes the image generator network weights of the first domain and the image generator network framework of the first domain. Among them, the image generator network framework of the first domain selects the StyleGAN2 network framework.

[0045] It should be noted that the image generator network in the first field and the image generator network in the second field use the same network structure framework, both of which are StyleGAN2 network frameworks, so that the image generator network in the second field can be obtained by performing transfer learning with the weights of the image generator in the first field as initialization. Among them, the image generator network in the second field is a generator network / generative adversarial network that generates images in the second field. The image generator network in the first field is a generator network / generative adversarial network that generates images in the first field. The StyleGAN2 network is an improved version of the StyleGAN network, which is a generative adversarial network that includes a generator and a discriminator.

[0046] Step S22: Using the image generator network of the first domain as the initial network, using the data set of the second domain to perform transfer learning on the initial network, and obtaining the image generator network of the second domain.

[0047] Specifically, the image generator network weights of the first domain are used as initialization to construct an image generator network of the second domain.

[0048] Before this step, it also includes: obtaining a data set of the second domain. This step specifically includes: using the weights of the image generator network of the first domain as initialization to perform transfer learning on the data set of the second domain to obtain an image generator network of the second domain.

[0049] Among them, the data set of the second domain is composed of multiple second domain images. Ways to obtain the data set of the second domain include: network screenshots, animation video screenshots, and game screenshots, etc., which are not limited here. Use a deep learning algorithm to perform face detection, key point detection, and face alignment processing on the second domain images, and filter out all second domain images with face images to form the data set. Among them, face alignment includes: aligning the second domain image in response to the face image being a side face image to a front face image, and the face image is a front face image. Finally, all images in the filtered data set are processed with a super-resolution algorithm to adjust the size and resolution of the image. Specifically, the resolution of the image can be adjusted to 1024×1024.

[0050] In a specific implementation, after manually collecting more than 100K second-domain pictures from web pages, the more than 100K second-domain pictures are subjected to face detection, key point detection, and face rotation and alignment processing by a deep learning algorithm, and finally 23K pictures with second-domain facial portraits are screened out to form a second-domain data set, and all pictures in the second-domain data set are adjusted to a resolution of 1024×1024 using a super-resolution algorithm.

[0051] This step specifically includes: using the image generator network of the first domain as the initial network, performing migration iterative learning on the initial network using a data set with a resolution of 1024×1024, and obtaining an image generator network of the second domain. All images in the data set are images with a resolution of 1024×1024.

[0052] It should be noted that transfer learning is to transfer the trained model parameters to the new model to help train the new model. In this embodiment, the StyleGAN2 network framework is used to construct an image generator network that generates images in the second domain with the weight parameters of the image generator network in the first domain as the initial values ​​and the data set in the second domain as the output. The image generator network in the second domain in this embodiment is different from the traditional generator network structure. The image generator in the second domain randomly samples a 512-dimensional z vector from a Gaussian distribution, and the z vector is then mapped to an 18×512-dimensional w latent vector space through a FC (fully connected) network. The w latent vector space has better separation attribute features than the z vector space, providing a basis for subsequent multi-attribute style control. For further information on the structure of the image generator network in the second domain in this embodiment, please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of an embodiment of the image generator network of the second domain in the present application. The generation of the image generator network of the second domain includes mapping the z vector to the w latent vector through a mapping network, inputting the weight of the first domain image generator network controlled by the w latent vector into the second domain image generator network structure for iterative training to obtain the image generator of the second domain, so that the image generator can output the second domain image. Figure 3 The arrows in include the operation steps for generating the second domain image and are not part of the second domain image generator structure.

[0053] Step S23: Replace some convolutional layers of the image generator network of the first domain with some convolutional layers corresponding to the image generator network of the second domain to generate the first image generator network.

[0054] In this embodiment, the image generator network in the first domain and the image generator network in the second domain have the same structural framework, and both use the StyleGAN2 network framework. The low-level convolution of the StyleGAN2 network framework is mainly responsible for geometric information such as face shape, expression, and hairstyle, and the high-level convolution mainly affects detail information such as color. Therefore, the conversion from one data domain to another can be achieved by combining the bottom convolution layer of the first domain and the high-level convolution layer of the second domain.

[0055] The specific exchange structure is as follows Figure 4 As shown, Figure 4This is a structural diagram of an implementation method of convolutional layer exchange in the present application. In the figure, a represents the convolutional layer weights of the image generator network in the first field. b represents the convolutional layer weights of the image generator network in the second field obtained by performing transfer iterative learning with the weights of the image generator network in the first field as the initialization, wherein the number of training iterations in the transfer iterative learning is less than 10K times. c represents replacing the high-level convolutional layer weights in the image generator network in the first field with the high-level convolutional layer weights from the image generator network in the second field, thereby reconstructing a new image generator network, that is, constructing the first image generator network in this embodiment. In this embodiment, the StyleGAN2 network includes a generator and a discriminator, which can automatically obtain the weights of the network convolutional layer, and discriminate whether the weights belong to the high-level convolutional layer weights or the bottom-level convolutional layer weights.

[0056] It should be noted that, in the iterative training process of generating the image generator network of the second domain, too many iterations will cause the constructed first image generation network to be unable to generate the second domain image due to the large difference in weight distribution between the image generator network of the first domain and the image generator network of the second domain, but too few iterations will affect the image generation quality. Therefore, in this embodiment, in order to ensure the image generation quality, the first domain image is converted into the initial image of the second domain by the first image generation network generated by low-iteration training, and then the initial image of the second domain is repaired by combining with the second image generation network generated by high-iteration training. Among them, the number of iterations of low-iteration training is less than 10K times, and the first image generation network is formed by exchanging the convolutional layer of the image generator network of the second domain obtained through low-iteration training with the convolutional layer of the image generator network of the first domain.

[0057] Step S13: Repair the initial image through the second image generation network to obtain an image corresponding to the original image in the second domain.

[0058] Wherein, the second image generation network is obtained through multiple iterative training of the image generator network of the second field. The image generator network of the second field is obtained by transfer learning the image generator network of the first field using the data set of the second field. Wherein, transfer learning includes iterative training. In this embodiment, the number of iterations of the image generator network of the second field is greater than the first set number and less than the second set number, wherein the first set number is zero or any positive integer greater than zero, and the second set number is 10K times or any positive integer less than 10K times. The number of iterations of the image generator network of the second field is less than 10K, which can ensure that the input of the first field image can output a better second field image. The second image generation network is obtained by multiple iterative transfer learning of the image generator network of the first field using the data set of the second field, and the number of iterative training of the second image generation network is more than the number of iterations of the image generator network of the second field.

[0059] In one embodiment, the image generator network of the second domain is obtained by training the StyleGAN2 network framework for less than 10K iterations using the weights of the image generator network of the first domain and the dataset of the second domain. The second image generation network is obtained by training the StyleGAN2 network framework for more than 10K iterations using the weights of the image generator network of the first domain and the dataset of the second domain.

[0060] It should be noted that the number of iterations is different, and the time consumed by the generation network is different. In this embodiment, the second image generation network takes longer than the image generator network in the second field, and the quality of the generated second field image is better and the image is clearer. However, directly using the second image generation network to convert the original image into an image in the second field will cause the generated image in the second field to be significantly different from the original image in the first field. The specific difference includes that the facial key points of the generated image in the second field and the original image in the first field are significantly different, which is not limited here. In this embodiment, the second image generation network is mainly responsible for repairing the defects in the second field image generated by the first image generation network, thereby improving the generation quality of the second field image.

[0061] In an application scenario, the first domain is a real person domain, and the second domain is a two-dimensional domain. The original image in the first domain is a real person face image uploaded by the anchor, and the image in the second domain is a face image in the two-dimensional domain. The image in the first domain is converted into an image in the second domain using the first image generation network and the second image generation network. For details, please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of an application scenario of the image generation network of this application. Figure 5As shown, the original image of the first field is input into the first image generator network (Latent-1) to convert the original image of the first field into the initial image of the second field, and then the initial image is input into the second image generator network (Latent-2) for repair, and the initial image is converted into the image corresponding to the second field. The first image generator network (Latent-1) is obtained by replacing some convolutional layers corresponding to the image generator network of the first field with some convolutional layers of the image generator network of the second field. The image generator network of the second field is obtained by transfer learning the image generator network of the first field using the data set of the second field, and the image generator network of the first field is obtained using the existing StyleGAN2 network. The second image generation network (Latent-2) is obtained by training the image generator network of the second field through multiple iterations. In one embodiment, in the process of generating the image of the second field using the second image generation network, the style attributes of the image of the second field can also be modified in combination with style fusion technology (Style-mixing) and truncation trick (truncation-trick), which is not limited here.

[0062] The beneficial effect of this embodiment is that the first domain image is converted into the second domain image through the first image generation network, and then the generated second domain image is repaired through the second image generation network, so as to obtain a better second domain image. Compared with the traditional image generation network, the second domain image generation system is optimized by the second order, which improves the generation quality of the second domain image and repairs the defects of the first domain image conversion into the second domain image.

[0063] This application also provides another image generation method, see Figure 6 , Figure 6 FIG. 1 is a flow chart of another embodiment of the image generation method of the present application. Figure 6 As shown, including:

[0064] Step S51: Acquire the original image of the first field.

[0065] The first domain is a real person domain, and the original image in the first domain is a real person domain image, including the anchor avatar uploaded by the anchor.

[0066] Step S52: Utilize the first image generation network to convert the original image into an initial image in the second domain.

[0067] Among them, the first image generation network is obtained by using the data set of the second field to perform transfer learning on the image generator network of the first field to obtain the image generator network of the second field, and replacing some convolutional layers of the image generator network of the second field with some convolutional layers corresponding to the image generator network of the first field.

[0068] Before step S52, the method also includes: performing face detection on the original image using a deep learning algorithm; in response to the detected face being a side face image, aligning the side face into a front face image; and converting the front face image into an initial image of the second domain.

[0069] In this embodiment, before obtaining the first image generation network, it also includes obtaining an image generator network in the second domain, and the image generator network in the second domain is used to generate a second domain image, and the second domain image is a two-dimensional domain image.

[0070] The step of obtaining the image generator network of the second domain includes the following steps, see Figure 7 , Figure 7 A flowchart of an implementation method of generating a second domain image generator network for this application is provided. It includes:

[0071] Step S71: Obtain an image generator network in the first domain and a data set in the second domain.

[0072] Among them, the image generator network of the first field uses the StyleGAN2 network architecture. In this application, the image generator network of the first field can be referred to as the first field image generator for generating first field images. The data set of the first field uses the FFHQ (high-quality face image data set) public data set generally recognized by the industry, and the weight of the image generator network of the first field is also derived from FFHQ.

[0073] The data set of the second domain includes second domain images, including a plurality of second domain images obtained from network screenshots, animation video screenshots, and game screenshots. Face detection, key point detection, and face alignment processing are performed on the obtained second domain images to screen out the data set of the second domain with face images. The face alignment processing includes aligning the side face image to a front face image in response to the detected face being a side face image.

[0074] Step S62: adjusting the data set by a super-resolution algorithm to obtain a data set with a set resolution.

[0075] The pixels of the second domain images in the dataset are resized to a resolution of 1024×1024.

[0076] Step S63: The image generator network of the first domain is used as the initial network, and a data set of a set resolution is used to perform transfer learning on the initial network to obtain an image generator network of the second domain.

[0077] Using the StyleGAN2 network framework, with the weight parameters of the image generator network in the first domain as the initial values ​​and the data set in the second domain as the output, iterative transfer learning training is performed to construct the image generator network in the second domain. For details, please refer to the description of step S12 in the above embodiment.

[0078] In this embodiment, there is no need to establish the first domain dataset and the second domain dataset in pairs. By using the public first domain dataset and the second domain dataset to train the second domain generator network, the time for establishing the dataset in pairs is saved, and the second domain dataset can also be applied to other applications with the same network structure.

[0079] Step S53: Repair the initial image through the second image generation network to obtain an image corresponding to the original image in the second domain.

[0080] The second image generation network is obtained by multiple iterative training of the image generator network in the second field. The number of iterative training of the second image generation network is greater than the number of iterations of the image generator network in the second field. The number of iterations of the image generator network in the second field is greater than the first set number of times and less than the second set number of times. In one embodiment, the number of iterations of the image generator network in the second field is greater than zero and less than 10K times.

[0081] Step S54: modify the style attributes of the image using style fusion technology and truncation techniques.

[0082] In this embodiment, the image generator network of the second domain is different from the traditional generator network structure. The image generator of the second domain includes randomly sampling a 512-dimensional z vector from a Gaussian distribution, and the z vector is then mapped to a 18×512-dimensional w latent vector space through a fully connected layer (FC) network. The feature dimension of the w latent vector space is 18×512, where the 1st to 7th layer features contain geometric information such as face shape, posture, hairstyle, and expression, and the 8th to 18th layer features represent detailed information such as background color, hair color, and two-dimensional domain style. Therefore, the attribute modification of the two-dimensional image can be achieved by controlling a specific latent vector. Specifically, it includes: randomly sampling a sample vector from a Gaussian distribution, and the sample vector is mapped to the latent vector space of w through the FC network and the latent vector of the image through style fusion technology (Style-mixing) and truncation-trick to achieve attribute modification of the feature dimension.

[0083] For more details, please refer to Figure 8 , Figure 8 FIG. 1 is a schematic diagram of a framework of an implementation method of a multi-style attribute control method of the present application. Figure 8As shown, an original image of a first domain is obtained, and the original image of the first domain is converted into an image of a second domain through a first image generation network and a second image generation network, and w latent vector space feature information of the image of the second domain is obtained, and a sample vector is randomly sampled from a Gaussian distribution to obtain w latent vector space feature information of the sample vector, wherein the w latent vector space feature information includes 1 to 18 layers of feature information (wherein 1 to 7 layers of features include geometric information such as face shape, posture, hairstyle, and expression, and 8 to 18 layers of features represent detail information such as background color, hair color, and two-dimensional domain style), and the style fusion technology (Style-mixing) is used to combine the 1-7 layers of w latent vectors of the generated second domain image with the 8-18 layers of w latent vectors of the randomly sampled sample to generate new w latent vector space feature information, and the new w latent vector space feature information is sequentially input into the space framework layer corresponding to the styleGAN network framework using the truncation technique to obtain a second domain image with multiple style attributes.

[0084] In this embodiment, step S54 includes: modifying the feature latent vector of the image generator network using style fusion technology and truncation techniques, thereby modifying the style attributes of the image. This step also includes obtaining a modification instruction from the user / anchor, wherein the modification instruction includes any one of modifying the background color, modifying the hair color, and modifying the style of the second domain, and modifying the feature latent vector according to the obtained modification instruction, thereby modifying the style attributes of the second domain image, and obtaining a plurality of two-dimensional images with different style attributes.

[0085] The beneficial effects of this embodiment are: based on the second-order optimization second domain image generation method in the above embodiment, a personalized second domain virtual image is generated according to the uploaded image for the anchor, and through the control of the feature latent vector, a plurality of second domain images with different style attributes are conveniently generated for the anchor to choose. In addition, the network training in this application does not need to use paired data and can be applied to image generation requirements of different styles.

[0086] This application also provides a smart terminal, see Fig. 9 , Fig. 9 This is a schematic diagram of the structure of an implementation scheme of the smart terminal of the present application. Fig. 9 As shown, the intelligent terminal 70 includes a processor 701 and a memory 702 coupled to each other, and the processor 701 is used to execute the program instructions stored in the memory 702 to implement the steps in any of the above method embodiments or the steps correspondingly executed by the client in any of the above method embodiments. In addition to the above processor 701 and memory 702, the terminal may also include a touch screen, a printing component, a communication circuit, etc. according to requirements, which are not limited here.

[0087] Specifically, the processor 701 is used to control itself and the memory 702 to implement the steps in any of the above method embodiments. The processor 701 can also be called a CPU (Central Processing Unit). The processor 701 may be an integrated circuit chip with signal processing capabilities. The processor 701 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 701 can be implemented by multiple integrated circuit chips.

[0088] The present application also provides a computer-readable storage medium, such as Fig.10 As shown, Fig.10 This is a schematic diagram of the structure of an embodiment of a computer-readable storage medium of the present application.

[0089] The computer-readable storage medium 80 includes a computer program 801 stored on the computer-readable storage medium 80. When the computer program 801 is executed by the above-mentioned processor, it implements the steps in any of the above-mentioned method embodiments or the steps corresponding to the error correction method for network timing in the above-mentioned method embodiments.

[0090] Specifically, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 80. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a computer-readable storage medium 80, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 80 includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0091] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0092] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0093] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0094] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0095] The above are only implementation methods of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image generation method, characterized in that: The image generation method comprises: Acquire the original image of the first field; The original image is converted into an initial image of a second domain by using a first image generation network; wherein the first image generation network is obtained by replacing some convolutional layers corresponding to the image generator network of the first domain with some convolutional layers of the image generator network of the second domain, and the image generator network of the second domain is obtained by performing transfer learning on the image generator network of the first domain using a data set of the second domain; the image generator network of the first domain is a StyleGAN2 network framework; The initial image is repaired by a second image generation network to obtain an image corresponding to the original image in the second domain; wherein the second image generation network is obtained by multiple iterative training of an image generator network in the second domain; wherein the second image generation network is obtained by multiple iterative training of an image generator network in the second domain; the image generator network in the second domain includes randomly collecting a 512-dimensional z vector from a Gaussian distribution, and using a fully connected layer network to map the z vector to a 18×512-dimensional w latent vector space.

2. The image generation method according to claim 1, characterized in that: The step of converting the original image into an initial image of the second domain using the first image generation network comprises: Performing face detection on the original image using a deep learning algorithm; In response to the detected human face being a side face image, aligning the side face into a front face image; The frontal face image is converted into an initial image of the second domain.

3. The image generation method according to claim 1, characterized in that: Before the step of acquiring the original image of the first field, the method includes: Obtaining an image generator network in the first domain and a dataset in the second domain; The data set is adjusted by a super-resolution algorithm to obtain a data set of a set resolution; The image generator network of the first domain is used as the initial network, and the initial network is subjected to migration iterative learning using the data set of the set resolution to obtain the image generator network of the second domain.

4. The image generation method according to claim 3, characterized in that: The data set includes second domain images; Before the step of adjusting the data set by a super-resolution algorithm to obtain a data set with a set resolution, the method further includes: Performing face detection on the second domain image using a deep learning algorithm; In response to the detected human face being a side face image, aligning the side face into a front face image; The frontal face images are screened out to form the data set.

5. The image generation method according to claim 3, characterized in that: The image generator of the first domain and the image generator of the second domain both include a convolution layer for collecting first facial features and a convolution layer for collecting second facial features; wherein the first facial features include at least one of face shape, expression and hairstyle, and the second facial features include at least color; After the step of using the image generator network of the first domain as the initial network and performing migration iterative learning on the initial network using the data set of the set resolution to obtain the image generator network of the second domain, the method further includes: A first image generation network is obtained by replacing the convolutional layer of the second facial feature in the first field with the convolutional layer of the second facial feature in the second field.

6. The image generation method according to claim 3, characterized in that: The number of iterations of the image generator network in the second field is greater than the first set number and less than the second set number.

7. The image generation method according to claim 5, characterized in that: The number of iterative training of the second image generation network is greater than the number of iterations of the image generator network of the second domain.

8. The image generation method according to claim 1, characterized in that: The first domain is a real person domain, and the second domain is a two-dimensional domain.

9. The image generation method according to claim 1, characterized in that: After the step of repairing the initial image by a second image generation network to obtain an image corresponding to the original image in the second domain, the method further includes: The style attributes of the image are modified using style fusion techniques and truncation techniques.

10. The image generation method according to claim 9, characterized in that: The w latent vector space feature dimension is 18 layers; wherein the 1-7 layers of latent vectors include collecting at least one feature of face shape, posture, hairstyle and expression, and the 8-18 layers of latent vectors include collecting at least one feature of background color, hair color and second domain style; The step of modifying the style attributes of the image by using the style fusion technology and the truncation technique comprises: The feature latent vector of the image generator network is modified using style fusion technology and truncation techniques to modify the style attributes of the image.

11. An intelligent terminal, characterized in that: The intelligent terminal comprises a processor and a memory coupled to each other, the memory is used to store program instructions, and the processor is used to execute the program instructions stored in the memory to implement the image generation method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises a processor and a memory, the memory stores computer program instructions, and the processor is used to execute the program instructions to implement the image generation method according to any one of claims 1 to 10.

Images