Model training and image processing method, device, equipment and storage medium

Abstract

Embodiments of the application disclose a model training and image processing method and device, equipment and a storage medium. The model comprises an encoder and a first decoder, the encoder comprises a full convolution network, and the model training method comprises the following steps: obtaining a content sample set, a style sample set and a real sample set, wherein the content samples in the content sample set are images with a first resolution, and there is a multiple relationship between the first resolution and a preset second resolution; obtaining, by using the encoder, a content feature of each content sample and a first style code corresponding to each content sample; generating, by using the first decoder, a style image with the second resolution and adjustable based on the content feature of each content sample and the first style code corresponding to the content feature, and obtaining a style image set; and iteratively training the model based on each style image in the style image set and a real sample corresponding to each style image until a convergence condition is met.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of computer vision technology, and in particular to a model training and image processing method, apparatus, device, and storage medium. Background Technology

[0002] Portrait editing refers to the generation and editing of input facial images, including facial stylization, super-resolution facial reconstruction, facial attribute editing, and generating facial images from line drawings. Currently, mainstream technologies employ image transformation methods, training networks to learn the mapping relationship between input face-related images and target faces. To achieve better results, a Style-Based Generator Architecture for Generative Adversarial Networks (StyleGAN) decoder is typically introduced to handle high-resolution portrait editing. However, the StyleGAN decoder is limited to processing fixed-size images and requires face alignment. Related techniques can only edit facial regions within a limited field of view in the input photo, failing to perform corresponding editing outside these regions, resulting in noticeable inconsistencies at the boundaries of the facial areas. Designing a universal image transformation framework to support high-resolution portrait editing with a standard field of view remains a challenge. Summary of the Invention

[0003] In view of this, embodiments of this application provide at least one model training and image processing method, apparatus, device, and storage medium.

[0004] In a first aspect, embodiments of this application provide a model training method. The model includes an encoder and a first decoder. The encoder includes a fully convolutional network. The method includes: acquiring a content sample set, a style sample set, and a set of real samples, wherein the content samples in the content sample set are images with a first resolution, and there is a multiple relationship between the first resolution and a preset second resolution; acquiring content features of each content sample and a first style code of a style sample corresponding to each content sample in the style sample set through the encoder; generating an adjustable style image with the second resolution through the first decoder based on the content features of each content sample and the first style code corresponding to the content features, and obtaining a style image set including the style images; and iteratively training the model based on each style image in the style image set and the real sample corresponding to each style image in the set of real samples until a convergence condition is met.

[0005] In some embodiments, the first decoder is a decoder reconstructed based on the StyleGAN decoder, which processes content features at a first non-fixed resolution in an equivalent manner to how the StyleGAN decoder processes content features at a fixed resolution, in order to generate a style image at a second non-fixed resolution.

[0006] In this way, the operation methods of content features under fixed resolution are applied to the content features under the first non-fixed resolution, making the operation of content features under the first non-fixed resolution equivalent to the operation of content features under the original fixed resolution. This cleverly reconstructs the StyleGAN decoder, so that the reconstructed StyleGAN decoder does not change the function and operation methods of the original StyleGAN decoder. It can process the content features corresponding to the aligned or misaligned images under the first non-fixed resolution, thereby generating aligned or misaligned images under the second non-fixed resolution.

[0007] In some embodiments, the encoder includes a first feature layer module and a transformation module. The step of obtaining content features for each content sample through the encoder includes: for each content sample, determining at least one intermediate feature layer in the first feature layer module based on the preset second resolution and the first resolution of the content sample; obtaining a first intermediate feature of a style image corresponding to the content sample through the at least one intermediate feature layer based on each content sample; and converting the first intermediate feature into a feature with the same number of channels as the first feature layer of the first decoder through the transformation module, thereby obtaining the content features for each content sample.

[0008] Thus, firstly, at least one intermediate feature layer is determined in the first feature layer module to obtain the first intermediate feature; then, the first intermediate feature is obtained by using the features output by at least one intermediate feature layer; finally, the first intermediate feature is converted into a feature with the same number of channels as the first feature layer of the first decoder through the conversion module, thereby obtaining the content feature of the content sample that is convenient for the first decoder to process.

[0009] In some embodiments, the first decoder includes a second feature layer module. The step of generating an adjustable style image with a second resolution based on the content features of each content sample and a first style code corresponding to the content features, through the first decoder, includes: determining a set of target feature layers in the first feature layer module with the same resolution as the first m feature layers in the second feature layer module; determining at least one target feature layer at each resolution in the set of target feature layers; inputting a second intermediate feature output by the target feature layer in the at least one target feature layer into at least one feature layer with the same resolution in the second feature layer module; and generating an adjustable style image with a second resolution based on the content features of each content sample, the style code corresponding to the content features, and the second intermediate feature input into the second feature layer module, through the first decoder.

[0010] In this way, by inputting the second intermediate feature output from the target feature layer in the first feature layer module into at least one feature layer with the same resolution in the second feature layer module, more structural information is provided to the style image generated by the first decoder, thereby enabling the generated style image to include more detailed information.

[0011] In some embodiments, determining the target feature layer set with the same resolution as the first m feature layers in the second feature layer module in the first feature layer module includes: determining the value of m based on the task content of generating the style image; and determining the target feature layer set with the same resolution as the first m feature layers in the second feature layer module in the first feature layer module.

[0012] In this way, based on the task content of generating style images, the value of m in the first m feature layers is determined respectively, so as to set the number of feature layers in the input second feature layer module for different task content, so as to better train the model to complete the task.

[0013] In some embodiments, iteratively training the model based on each style image in the generated style image set and the real sample corresponding to each style image until a convergence condition is met includes: determining a loss function for training the model based on the task content that generates the style images; determining a loss value corresponding to the loss function based on each style image in the generated style image set and the real sample corresponding to each style image; and completing the training of the model when the loss value meets the convergence condition.

[0014] In this way, different loss functions are determined for different task content; then, by determining the loss value corresponding to the loss function, the model is trained so that the model can meet the convergence condition.

[0015] In some embodiments, when the task of generating the style image includes image encoding / decoding or generating a face image from a line drawing, the loss function includes a reconstruction loss, wherein the reconstruction loss is used to characterize the similarity between the generated style image and the corresponding real sample; and / or, when the task of generating the style image includes face super-resolution reconstruction, the loss function includes the reconstruction loss and an adversarial loss, wherein the adversarial loss is the adversarial loss of the first decoder; and / or, when the task of generating the style image is video face attribute editing or video face stylization, the loss function includes the reconstruction loss, the adversarial loss, and a temporal continuity loss, wherein the temporal continuity loss is used to characterize the consistency of semantically identical pixels in the style images generated from different image frames in the video.

[0016] In this way, by classifying the task content of generating style images and setting different loss functions for different task content, the model can be trained better and its robustness can be improved.

[0017] Secondly, embodiments of this application provide an image processing method applied to a model including an encoder and a first decoder. The encoder includes a fully convolutional network. The method includes: acquiring a content image to be processed with a first resolution, a style example matching the content image, and a preset second resolution, wherein there is a multiple relationship between the first resolution and the second resolution; acquiring content features of the content image and a first style encoding of the style example through the encoder; and generating an adjustable style image with the second resolution through the first decoder based on the content features and the first style encoding.

[0018] In some embodiments, the encoder includes a first feature layer module and a conversion module. The first style encoding of obtaining the content features of the content image and the style sample through the encoder includes: determining at least one intermediate feature layer in the first feature layer module based on the preset second resolution and the first resolution of the content sample; obtaining a first intermediate feature of the style image through the at least one intermediate feature layer based on the content image; and converting the first intermediate feature into a feature with the same number of channels as the first feature layer of the first decoder through the conversion module, based on the first intermediate feature, to obtain the content features of the style image.

[0019] In some embodiments, the first decoder includes a second feature layer module. The step of generating an adjustable style image with a second resolution based on the content features and the first style encoding via the first decoder includes: determining a set of target feature layers in the first feature layer module with the same resolution as the first m feature layers in the second feature layer module; determining at least one target feature layer at each resolution in the set of target feature layers; inputting a second intermediate feature output by the target feature layer in the at least one target feature layer into at least one feature layer with the same resolution in the second feature layer module; and generating an adjustable style image with a second resolution via the first decoder based on the content features, the first style encoding, and the second intermediate feature input into the second feature layer module.

[0020] In some embodiments, determining the target feature layer set with the same resolution as the first m feature layers in the second feature layer module in the first feature layer module includes: determining the value of m based on the task content of generating the style image; and determining the target feature layer set with the same resolution as the first m feature layers in the second feature layer module in the first feature layer module.

[0021] In some embodiments, when the task of generating the style image includes image encoding / decoding or face super-resolution reconstruction, the step of obtaining the content image to be processed with a first resolution and the style example matching the content image includes: obtaining the first image to be processed with a first resolution; cropping and obtaining an aligned second image with a preset size based on the first image; using the first image as the content image and the second image as the style example.

[0022] In some embodiments, where the task of generating the style image includes image encoding and decoding, after obtaining the first style code of the style sample through the encoder, the method further includes: editing the first style code to obtain an edited second style code; the step of generating an adjustable style image with the second resolution through the first decoder based on the content features and the first style code includes: generating an adjustable style image with the second resolution through the first decoder based on the content features and the second style code.

[0023] In some embodiments, when the task of generating the style image includes generating a face image based on a line drawing, the step of obtaining a content image to be processed with a first resolution and a style example matching the content image includes: obtaining a line drawing image to be processed and a third image with a style to be generated; cropping and obtaining a fourth image and a fifth image with a preset size based on the line drawing image and the third image respectively; using the line drawing image as the content image, and using both the fourth image and the fifth image as the style example; correspondingly, the step of obtaining a first style code of the style example through the encoder includes: inputting the fourth image into the encoder to obtain a third style code for influencing the structure of the style image; inputting the fifth image into the encoder to obtain a fourth style code for influencing the color of the style image; concatenating the third style code and the fourth style code to obtain the first style code of the style example.

[0024] In some embodiments, when the task of generating the style image includes video face attribute editing or video face stylization, the step of obtaining the content image to be processed with a first resolution and the style example matching the content image includes: obtaining the video to be processed; for each frame image in the video, cropping to obtain an aligned sixth image with a preset size; using each frame image in the video as the content image, and using the sixth image corresponding to each frame image as the style example.

[0025] In some embodiments, where the task of generating the style image includes editing facial attributes in a video, the method further includes: obtaining an edit vector for editing facial attributes in the video; correspondingly, after obtaining the first style code of the style example through the encoder, the method further includes: adding the first style code and the edit vector to obtain a fifth style code; generating an adjustable style image with the second resolution through the first decoder based on the content features and the first style code includes: generating an adjustable style image with the second resolution through the first decoder based on the content features and the fifth style code.

[0026] In some embodiments, when the task content for generating the style image includes video face stylization, the step of generating an adjustable style image with the second resolution through the first decoder based on the content features and the first style encoding includes: generating an adjustable style image with the second resolution through a second decoder based on the content features and the first style encoding, wherein the second decoder is a decoder obtained by fine-tuning the first decoder using a set of style samples.

[0027] Thirdly, embodiments of this application provide a model training apparatus. The model training model includes an encoder and a first decoder. The encoder includes a fully convolutional network. The model training apparatus includes: a first acquisition module, used to acquire a content sample set, a style sample set, and a real sample set, wherein the content samples in the content sample set are images with a first resolution, and there is a multiple relationship between the first resolution and a preset second resolution; a second acquisition module, used to acquire the content features of each content sample and the first style code of the style sample corresponding to each content sample in the style sample set through the encoder; a first generation module, used to generate an adjustable style image with the second resolution through the first decoder based on the content features of each content sample and the first style code corresponding to the content features, and obtain a style image set including the style images; and a training module, used to iteratively train the model based on each style image in the style image set and the real sample corresponding to each style image in the real sample set until the convergence condition is met.

[0028] Fourthly, embodiments of this application provide an image processing apparatus applied to a model including an encoder and a first decoder. The encoder includes a fully convolutional network. The image processing apparatus includes: a third acquisition module for acquiring a content image to be processed with a first resolution, a style example matching the content image, and a preset second resolution, wherein there is a multiple relationship between the first resolution and the second resolution; a fourth acquisition module for acquiring content features of the content image and a first style code of the style example through the encoder; and a second generation module for generating an adjustable style image with the second resolution based on the content features and the first style code through the first decoder.

[0029] Fifthly, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0030] Sixthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0031] In related technologies, decoders such as StyleGAN can only process images of fixed size, and the processed images require face alignment. The model provided in this application includes an encoder and a first decoder, the encoder comprising a fully convolutional network. First, based on the translation invariance of the fully convolutional network and the reusability of the convolutional kernels, the encoder can output content features at variable resolutions while outputting style codes. Second, the first decoder can be a decoder that generates a second non-fixed resolution image based on the content features reconstructed from the StyleGAN decoder, which can be used as input for the first non-fixed resolution content features. This allows for the generation and editing of aligned or misaligned images at non-fixed resolutions, thus solving the problem in related technologies that can only process fixed-size images and require face alignment. Finally, the encoder provided in this application can simultaneously encode content features and style codes, and can be trained end-to-end, thus better completing the learning task.

[0032] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0034] Figure 1A This is a schematic diagram of the structure of the StyleGAN model;

[0035] Figure 1B A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0036] Figure 2 A schematic diagram of the original StyleGAN decoder and the reconstructed StyleGANEX decoder provided for embodiments of this application;

[0037] Figure 3A A schematic diagram illustrating the implementation process of step S102 provided in an embodiment of this application;

[0038] Figure 3B A schematic diagram illustrating the implementation process of step S103 provided in an embodiment of this application;

[0039] Figure 4 This is a schematic diagram of an encoder-decoder structure provided in an embodiment of this application;

[0040] Figure 5 A schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application;

[0041] Figure 6 This is a schematic diagram of the composition structure of a model training device provided in an embodiment of this application;

[0042] Figure 7 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0045] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0047] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0048] The Adaptive Instance Normalization (AdaIN) layer is a network module. This module normalizes image features along the channel dimension, so that the mean and variance of the image features in each channel are normalized to between 0 and 1. Then, their mean and variance are adjusted to the mean and variance of the target style, thereby achieving style transfer.

[0049] StyleGAN is a style-based generative network that can map random style codes onto human face images.

[0050] Face images with a normal field of view: Face images with a normal field of view, such as ID photos and selfies, contain more background information, complete hairstyles, and part of the upper garment, and are more widely used in daily life. However, StyleGAN can only process face images with a limited field of view that are aligned, resulting in incomplete hair parts and only the collar part in the processed image, making it unsuitable for processing face images with a normal field of view.

[0051] Before introducing the embodiments of this disclosure, let's first introduce the StyleGAN model, such as... Figure 1A As shown, model 10 includes a mapping network 11 and a generator network 12, wherein:

[0052] The mapping network 11 can map the style code of the input content image I (the corresponding style code can be obtained by the encoder 111) or the randomly sampled style code z into an intermediate style code (which can be understood as an intermediate vector). The intermediate style code will then be passed to the generator network 12 as 18 control vectors of the generator network 12, so that different elements of the control vector can control different visual features.

[0053] Mapping network 11 consists of 8 fully connected (FC) layers, and the intermediate style code output by mapping network 11... The input size is the same as that of the StyleGAN model. Typically, the input size can be (512×1) or (512×18), meaning one 512-dimensional vector or 18 vectors of 512 dimensions each. If the input is a 512-dimensional vector, the input layer will copy it into 18 identical vectors.

[0054] The generator network 12 comprises nine generator modules that transform from 4*4 to 1024*1024. Specifically, each module corresponds to a 4*4, 8*8, 16*16, 32*32, 64*64, 128*128, 256*256, 512*512, and 1024*1024 transformation. Each module, except for the first and last generator modules, is subject to two control vectors (…). Figure 1A The intermediate style encoding (AdaIN adaptive instance normalization layer) influences the input of the affine transformation A. One control vector influences it once after upsampling, and another control vector influences it once after convolution. Both influences are achieved using the style adjustment module. The first module only has convolution operations, not upsampling, and therefore uses only one control vector. The last module, in addition to the control vectors corresponding to the upsampling and convolution operations, also includes a control vector corresponding to the transformation of the features into a 3-channel image. Therefore, intermediate style encoding... In total, they were transformed into 18 control vectors and passed to the generator network 12.

[0055] To put it another way, the first generation module includes an affine transformation module, a convolutional layer, and a style adjustment module, but the first generation module does not have an upsampling module; the other generation modules include an upsampling module, an affine transformation module, a convolutional layer, and a style adjustment module.

[0056] The implementation process of style adjustment in the first generation module is as follows: Figure 1A As shown: Encoding intermediate styles After passing through a learnable affine transformation module A (which can be a fully connected layer), the transformation is expanded by a style adjustment module to include a scaling factor y_(s,i) and a bias factor y_(b,i). The scaling factor is multiplied by the normalized features output from the convolutional layer, and then summed with the bias factor, thus completing one intermediate style encoding step. The process that influences the style of an image.

[0057] The lower the resolution of the layers in StyleGAN, the coarser the features it affects. These features can be divided into two types: 1) Coarse features—resolution no higher than 32*32, corresponding to layers 1 to 7, affecting pose, hairstyle, facial shape, facial features, and whether the eyes are open or closed; 2) Fine features—resolution from 64*64 to 1024*1024, corresponding to layers 8 to 18, affecting color (eyes, hair, and skin) and microscopic features.

[0058] StyleGAN can only process fixed-size images and requires face alignment, making it unsuitable for video applications. To address these issues, this application provides a model including an encoder and a first decoder, where the encoder comprises a fully convolutional network. Leveraging the translation invariance and reusability of convolutional kernels in a fully convolutional network, the encoder can output content features at variable resolutions while simultaneously outputting style codes. Furthermore, the first decoder can be a decoder that generates a second non-fixed-resolution image based on the content features reconstructed from the StyleGAN decoder, which can be used as input to the first non-fixed-resolution image. This enables the generation and editing of aligned or misaligned images at non-fixed resolutions.

[0059] This application provides a model training method that can be executed by a processor of a computer device. The computer device refers to a device with data processing capabilities, such as a server, laptop, tablet, desktop computer, smart TV, set-top box, or mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device). Figure 1BThis is a schematic diagram illustrating the implementation process of a model training method provided in an embodiment of this application, as shown below. Figure 1B As shown, the method includes the following steps S101 to S104:

[0060] Step S101: Obtain a content sample set, a style sample set, and a real sample set, wherein the content samples in the content sample set are images with a first resolution, and there is a multiple relationship between the first resolution and a preset second resolution.

[0061] Here, the content sample set can be images that provide content during image generation and editing. The style sample set can be a set of real image data or a set of artistic image data. Artistic images in the artistic image data set can be artistic portraits, artistic landscapes, cartoon images, comics, etc. For example, images in the artistic image data set can be images obtained through rendering and / or distortion techniques. The real sample set can be a set of real image data, such as real human face data, real animal data, real landscape data, etc., obtained through photography.

[0062] In some embodiments, there is a correspondence between content samples in the content sample set, style samples in the style sample set, and real samples in the real sample set. For example, content sample D and style sample B that match content sample D are used to generate a style image through the above model. The real image of this style image is real sample C. Then, content sample D, style sample B, and real sample C are three samples of the same style image. There is a correspondence between content sample D, style sample B, and real sample C, and they can form a sample group.

[0063] The preset second resolution can refer to the resolution of the style image generated by the above model. That is, the resolution of the generated style image can be preset and set according to the user's needs. For example, if the user needs to generate a 256*256 image, the preset second resolution is 256*256; if the user needs to generate a 1024*1024 image, the preset second resolution is 1024*1024. There is a multiple relationship between the first resolution and the preset second resolution, meaning there is a certain multiple relationship between the resolution of the content sample and the resolution of the style image generated by the model. If the multiple relationship between the resolution of the obtained original content sample and the preset second resolution is incorrect, the content sample can be obtained by upsampling or downsampling the original content sample, so that the content sample has a first resolution that is a multiple relationship with the preset second resolution. In some embodiments, the content sample can be an aligned or misaligned image; this application does not limit this.

[0064] Step S102: Obtain the content features of each content sample and the first style code of the style sample corresponding to each content sample in the style sample set through the encoder.

[0065] Here, since the encoder includes a fully convolutional network, it can process aligned or misaligned content samples at arbitrary resolutions, while outputting content features at variable resolutions.

[0066] In some embodiments, step S102 can be implemented by first inputting a content sample into the encoder to obtain the content features of the content sample; then inputting a style sample into the encoder to obtain a first style code; or by first inputting a style sample into the encoder to obtain a first style code; then inputting a content sample into the encoder to obtain the content features of the content sample. In some embodiments, if the content sample and the style sample are the same sample, step S102 can be implemented by inputting the same sample into the encoder to obtain both the content features and the first style code. This application does not limit the method for obtaining the content features and the first style code.

[0067] In some embodiments, the encoder may include multiple feature layers for multiple resolutions. Therefore, content features at the desired resolution can be obtained by selecting intermediate feature layers of the output content features in the encoder. For methods of obtaining first style encoding through the encoder in some embodiments, see related technologies.

[0068] Step S103: Based on the content features of each content sample and the first style code corresponding to the content features, generate an adjustable style image with the second resolution through the first decoder, and obtain a style image set including the style image.

[0069] Here, since the second resolution is adjustable, the resolution of the generated style image can be non-fixed. Since there is a multiple relationship between the first and second resolutions, the first resolution of the content sample also needs to be adjusted when the second resolution is adjusted. For example, if the multiple relationship between the first and second resolutions is that the second resolution is four times the first resolution, then when the second resolution is 4H*4W, the first resolution is H*W. This application does not limit the method by which the first decoder generates a style image with a second resolution.

[0070] In some embodiments, the first decoder can be a decoder reconstructed based on the StyleGAN decoder. It can process content features at a first non-fixed resolution using an equivalent method to how the StyleGAN decoder processes content features at a fixed resolution, to generate a style image at a second non-fixed resolution. Here, the non-fixed resolution can be any resolution. In some embodiments, the first and second non-fixed resolutions can be multiples of each other.

[0071] Typically, the StyleGAN decoder takes input features at a fixed resolution of 4x4x512 and generates a style image at a fixed resolution of 1024x1024. As mentioned above, the StyleGAN decoder consists of 18 layers. The first layer has a resolution of 4x4, layers 2 and 3 have a resolution of 8x8, layers 4 and 5 have a resolution of 16x16, layers 6 and 7 have a resolution of 32x32, layers 8 and 9 have a resolution of 64x64, layers 10 and 11 have a resolution of 128x128, layers 12 and 13 have a resolution of 256x256, layers 14 and 15 have a resolution of 512x512, and layers 16 to 18 have a resolution of 1024x1024. The first 7 layers influence the coarse features (structural features) of the generated style image, while the last 11 layers influence the fine features (color features).

[0072] To enable the reconstructed StyleGANEX decoder to process content features at a first non-fixed resolution in an equivalent manner to how the original StyleGAN decoder processes content features at a fixed resolution, in some embodiments, the following can be addressed: Figure 1AThe original StyleGAN decoder shown removes upsampling operations from the first 7 layers and adjusts the convolutions in each layer to corresponding dilated convolutions to match the receptive field. In implementation, dilated convolutions can be obtained by changing the dilation rate of each convolutional kernel. The dilation rate defines the spacing between values ​​processed by the convolutional kernel, i.e., the number of intervals between points in the convolutional kernel. The normal convolutional kernel has a dilation rate of 1. Increasing the dilation rate expands the receptive field of the convolutional neural network; for example, a 3x3 convolutional kernel with a dilation rate of 2 can achieve the same receptive field as a 5x5 convolutional kernel. The dilation rate of layer 1 is 8, layers 2-3 have a dilation rate of 4, layers 4-5 have a dilation rate of 2, layers 6-7 have a dilation rate of 1, and the structure from layers 8-9 onwards remains unchanged. This achieves equivalence between operations on content features at the first non-fixed resolution and operations on content features at the original fixed resolution, without altering the original StyleGAN decoder functionality, by changing the size of the convolutional kernel. In other words, the operation methods for content features at a fixed resolution are applied to content features at a first non-fixed resolution, making the operation of content features at the first non-fixed resolution equivalent to the operation of content features at the original fixed resolution, thus cleverly reconstructing the StyleGAN decoder.

[0073] The following section, combining the structures of the StyleGAN decoder and the StyleGANEX decoder, will further explain the method of the StyleGANEX decoder. Figure 2 The structures of the StyleGAN decoder and StyleGANEX decoder are shown. It can be seen that the StyleGANEX decoder has an input feature resolution of H*W*512. The first 7 layers have no upsampling and all have a resolution of H*W. Layers 8 and 9, after upsampling by the upsampling module, have a resolution of 2H*2W. From layers 8 to 9 onwards, all other layers undergo upsampling, with the resolution doubling each time. The StyleGAN decoder, on the other hand, begins upsampling from layers 2 and 3. In other words, from layers 8 to 9 onwards, the structures of the StyleGAN decoder and StyleGANEX decoder are the same; the difference lies in the first 7 layers.

[0074] The following explanation uses layers 4 and 5 as an example to illustrate the modification method for the first 7 layers. The resolution of both the StyleGAN decoder and the StyleGANEX decoder in layers 6 and 7 is H*W. The StyleGAN decoder in layers 4 and 5 has undergone upsampling, while the StyleGANEX decoder has not. In order to make their receptive fields the same, the expansion rate of layers 4 and 5 of the StyleGANEX decoder can be set to 2. Similarly, the expansion rate of layers 2 and 3 of the StyleGANEX decoder is 4, the expansion rate of layer 1 of the StyleGANEX decoder is 8, and the expansion rate of layers 6 and 7 of the StyleGANEX decoder is 1. Thus, the StyleGANEX decoder does not change the function and operation method of the StyleGAN decoder.

[0075] It should be noted that since the first 7 layers mainly affect the structural features of the generated style image, and in order to enable the reconstructed StyleGAN decoder to handle content features at arbitrary resolutions, it focuses more on the structural features of the image. Therefore, the dilation rate of the first 7 layers was modified here. Also, since the StyleGAN decoder itself is a fully convolutional network, it can handle misaligned images. Therefore, the reconstructed StyleGAN decoder can handle the content features corresponding to aligned or misaligned images at non-fixed resolutions, thereby generating aligned or misaligned images at non-fixed resolutions.

[0076] When the first decoder is the reconstructed StyleGAN decoder, the resolution of the content features input to the first decoder is H*W, and the resolution of the generated style image is 32H*32W. Therefore, the second non-fixed resolution is 32 times the first non-fixed resolution, and the resolution of the content features obtained by the encoder in step S102 can be H*W to facilitate input to the first decoder.

[0077] Step S104: Based on each style image in the style image set and the real samples corresponding to each style image in the real sample set, iteratively train the model until the convergence condition is met.

[0078] In some embodiments, the implementation of step S104 may include the following steps S1041 to S1043:

[0079] Step S1041: Based on the task content of generating the style image, determine the loss function used to train the model.

[0080] Here, the task of generating style images may include: image encoding and decoding, face super-resolution reconstruction, generating face images from line drawings, video face attribute editing or video face stylization, etc. This application embodiment does not limit the task of generating style images.

[0081] Typically, the loss function for training a model is given by the following formula (1):

[0082] L = L rec (x′,x)+L other (x′) (1);

[0083] Among them, L rec It is the reconstruction loss, used to characterize the similarity between the generated style image x′ and the corresponding real sample x. Commonly used metrics include L1 distance, L2 distance, perceptual distance, and identity similarity. other It is a loss function related to the task content of generating style images, and can have different definitions depending on the task content.

[0084] The following sections explain the components of the loss function based on the specific task requirements.

[0085] When the task of generating the style image includes image encoding / decoding or generating a face image from a line drawing, the loss function includes a reconstruction loss, where the reconstruction loss characterizes the similarity between the generated style image and the corresponding real sample; that is, when the task includes image encoding / decoding or generating a face image from a line drawing, the loss function in formula (1) does not include L. other Only L rec .

[0086] And / or, when the task content of generating style images includes face super-resolution reconstruction, the loss function includes reconstruction loss and adversarial loss, where the adversarial loss is the adversarial loss of the first decoder; that is, when the task content includes face super-resolution reconstruction, L in the loss function of formula (1) other To combat the losses.

[0087] And / or, when the task content of generating style images is video face attribute editing or video face stylization, the loss function includes reconstruction loss, adversarial loss, and temporal continuity loss. The temporal continuity loss characterizes the consistency of semantically identical pixels in style images generated from different image frames in the video, such as the consistency of pixels semantically representing the eyes in the style image. That is, when the task content includes video face attribute editing or video face stylization, L in the loss function of formula (1) other To combat loss and temporal continuity loss.

[0088] In this embodiment, the task content of generating style images is classified, and different loss functions are set for different task content in order to better train the model and improve its robustness.

[0089] Step S1042: Based on each style image in the generated style image set and the real sample corresponding to each style image, determine the loss value corresponding to the loss function;

[0090] Step S1043: If the loss value satisfies the convergence condition, the training of the model is completed.

[0091] Here, after the model is trained, the style images generated by the model are very close to the real samples.

[0092] In this embodiment, different loss functions are determined for different task content; then, by determining the loss value corresponding to the loss function, the model is trained so that the model can meet the convergence condition.

[0093] In related technologies, decoders such as StyleGAN can only process images of fixed size, and the processed images require face alignment. The model provided in this application includes an encoder and a first decoder, the encoder comprising a fully convolutional network. First, based on the translation invariance of the fully convolutional network and the reusability of the convolutional kernels, the encoder can output content features at variable resolutions while outputting style codes. Second, the first decoder can be a decoder that generates a second non-fixed resolution image based on the content features reconstructed from the StyleGAN decoder, which can be used as input for the first non-fixed resolution content features. This allows for the generation and editing of aligned or misaligned images at non-fixed resolutions, thus solving the problem in related technologies that can only process fixed-size images and require face alignment. Finally, the encoder provided in this application can simultaneously encode content features and style codes, and can be trained end-to-end, thus better completing the learning task.

[0094] In some embodiments, the encoder includes a first feature layer module and a transformation module, such as Figure 3A As shown, the implementation of "obtaining the content features of each content sample through the encoder" in step S102 may include the following steps S1021 to S1023:

[0095] Step S1021: For each content sample, based on the preset second resolution and the first resolution of the content sample, determine at least one intermediate feature layer in the first feature layer module.

[0096] Here, at least one intermediate feature layer is used to acquire the first intermediate feature of the generated content feature. Since the second resolution of the style image generated by the first decoder is preset, and the ratio between the resolution of the content feature input to the first decoder and the second resolution of the generated style image is fixed, the resolution of the content feature acquired by the encoder can be determined when the preset second resolution is obtained. Since the acquisition of the first intermediate feature is for obtaining the content feature, in some embodiments, the resolution of the first intermediate feature can be the same as the resolution of the content feature. That is, the resolution of the first intermediate feature can be obtained when the preset second resolution is obtained. Furthermore, since the encoder includes a first feature layer module, and the first feature layer module includes feature layers under multiple resolutions, the intermediate feature layer used to acquire the first intermediate feature can be determined in the first feature layer module when the first resolution of the content sample is obtained.

[0097] For example, if the preset second resolution is 4H*4W, the ratio between the resolution of the content features input to the first decoder and the second resolution of the generated style image is: the second resolution of the generated style image is 32 times the resolution of the input content features. Therefore, the resolution of the content features obtained by the encoder is H / 8*W / 8, meaning the resolution of the first intermediate feature is H / 8*W / 8. Since the first resolution is H*W, the intermediate feature layer is the feature layer corresponding to an 8x downsampling.

[0098] In some embodiments, the number of intermediate feature layers can be one or at least two, meaning that there are multiple feature layers corresponding to the same resolution.

[0099] Step S1022: Based on each content sample, obtain the first intermediate feature of the style image corresponding to the content sample through the at least one intermediate feature layer.

[0100] Here, when there is only one intermediate feature layer, the feature output by the intermediate feature layer is the first intermediate feature of the corresponding style image. When there are at least two intermediate feature layers, the feature obtained by concatenating the features output by each intermediate feature layer can be used as the first intermediate feature of the corresponding style image.

[0101] Step S1023: Based on the first intermediate feature, the first intermediate feature is converted into a feature with the same number of channels in the first feature layer of the first decoder by the conversion module, so as to obtain the content feature of each content sample.

[0102] Here, since the first intermediate feature obtained through step S1023 is a feature processed by the encoder, the number of channels of this feature may be different from the number of channels of the first feature layer of the first decoder, causing the first decoder to be unable to process it. Therefore, the first intermediate feature can be converted into a feature with the same number of channels as the first feature layer of the first decoder through the conversion module, thereby obtaining the content feature of each content sample that the first decoder can process. In some embodiments, the conversion module may include a convolutional layer.

[0103] In this embodiment of the application, firstly, at least one intermediate feature layer for obtaining the first intermediate feature is determined in the first feature layer module; then, the first intermediate feature is obtained by using the features output by at least one intermediate feature layer; finally, the first intermediate feature is converted into a feature with the same number of channels as the first feature layer of the first decoder by the conversion module, thereby obtaining the content feature of the content sample that the first decoder can process.

[0104] In some embodiments, the first decoder includes a second feature layer module, such as Figure 3B As shown, the implementation of step S103, "based on the content features of each content sample and the first style encoding corresponding to the content features, and through the first decoder, generating an adjustable style image with the second resolution," may include the following steps S1031 to S1034:

[0105] Step S1031: In the first feature layer module, determine a set of target feature layers with the same resolution as the first m feature layers in the second feature layer module.

[0106] Here, m can be any number less than or equal to the number of feature layers in the encoder. For example, if there are 13 feature layers in the encoder, then m can be any integer less than or equal to 13, such as 0, 1, or 7.

[0107] In some embodiments, the implementation of step S1031 may include the following steps S201 and S202:

[0108] Step S201: Determine the value of m based on the task content of generating the style image.

[0109] Here, since the features output by different feature layers in the second feature layer module have different effects on the generated style image, and the task content of generating the style image is different, the parts that need to affect the style image are also different. Therefore, the number of layers that need to be input from the first feature layer module to the second feature layer module are also different.

[0110] For example, when the task of generating style images involves image encoding and decoding, the goal is to reconstruct the realistic image using only content features and style encoding to simplify the processing. Therefore, it's unnecessary to input the second intermediate features from the first feature layer module into the second feature layer module. Furthermore, since the style encoding is typically edited after acquisition to alter facial attributes, inputting the second intermediate features from the first feature layer module into the second feature layer module—editing only the style encoding without modifying the input second intermediate features—would lead to output problems. Therefore, when the task of generating style images involves image encoding and decoding, the value of m can be 0.

[0111] For example, in the case where the task of generating style images is face super-resolution reconstruction, more structural features need to be obtained. Considering that the first 7 layers affect the structural features of the generated style images, the value of m can be 7.

[0112] For example, in the case where the task of generating style images is to generate face images based on line drawings, the value of m can be adjusted according to the level of detail of the line drawings. If the line drawings are more detailed, the value of m can be larger to retain more of the line drawing structure. Conversely, if the line drawings are coarser, the value of m can be smaller to optimize the line drawing structure as much as possible.

[0113] For example, when the task of generating style images involves editing or stylizing video face attributes, since videos contain a lot of information but require relatively few modifications, it's beneficial to transmit as much video content as possible. This allows for better editing of the video content and improves accuracy. Therefore, the value of m can be larger; for instance, if the encoder includes 13 feature layers, then the value of m can be 13.

[0114] Step S202: In the first feature layer module, determine a set of target feature layers with the same resolution as the first m feature layers in the second feature layer module.

[0115] In this embodiment, the value of m in the first m feature layers is determined according to the task content of generating style images, so as to set the number of feature layers in the input second feature layer module for different task content, so as to better train the model to complete the task.

[0116] Step S1032: In the set of target feature layers, determine at least one target feature layer at each resolution.

[0117] Here, since the number of feature layers at each resolution may be at least one, i.e., there is a case of greater than or equal to 2, the target feature layer determined in step S1032 may be all feature layers at each resolution, or it may be a partial feature layer at each resolution. For example, the feature layer with the deepest network depth may be selected as the target feature layer. This application embodiment does not limit the method of selecting the target feature layer.

[0118] Step S1033: Input the second intermediate feature output by the target feature layer in the at least one target feature layer into at least one feature layer with the same resolution in the second feature layer module.

[0119] Here, the number of feature layers at each resolution in the second feature layer module may also be at least one, that is, there may be a case where it is greater than or equal to 2. In this case, the implementation of step S1033 may include the following situations:

[0120] The first scenario: The second intermediate layer output by multiple target feature layers with the same resolution is input into the feature layers with the same resolution in multiple second feature layer modules (one-to-one correspondence);

[0121] The second scenario: Input the second intermediate layer output by a target feature layer into feature layers with the same resolution in multiple second feature layer modules;

[0122] The third scenario: Input the second intermediate layer output by a target feature layer into a feature layer with the same resolution in a second feature layer module.

[0123] The embodiments of this application do not limit the method of inputting the second intermediate feature output by the target feature layer into the feature layer with the same resolution in the second feature layer module.

[0124] Step S1034: Based on the content features of each content sample, the style encoding corresponding to the content features, and the second intermediate features input to the second feature layer module, an adjustable style image with the second resolution is generated through the first decoder.

[0125] Here, the implementation of step S1034 can include the following two cases:

[0126] The first scenario: For the feature layer that inputs the second intermediate feature in the second feature layer module: First, generate the third intermediate feature based on the content feature and style encoding of the first feature layer. Then, fuse the third intermediate feature with the second intermediate feature input to the first feature layer to obtain the output feature of the first feature layer. Starting from the second feature layer, generate the third intermediate feature based on the output feature of the previous feature layer and the style encoding of the current feature layer. Then, fuse the third intermediate feature with the second intermediate feature input to the current feature layer to obtain the output feature of the current feature layer.

[0127] The second scenario: For feature layers that do not have a second intermediate feature input: the output feature of the current feature layer is generated based on the output feature of the previous feature layer and the style encoding of the current feature layer. The last feature layer can output an adjustable style image with a second resolution. Implementation details can be found in related technologies.

[0128] In this embodiment of the application, by inputting the second intermediate feature output by the target feature layer in the first feature layer module into at least one feature layer with the same resolution in the second feature layer module, more structural information is provided to the style image generated by the first decoder, thereby making the generated style image include more detailed information.

[0129] This application addresses the limitations of existing StyleGAN-based portrait image editing methods, which are restricted to fixed-size images and require face alignment. These methods can only process the limited facial regions within the input field of view and cannot effectively handle input facial images and videos with a normal field of view. A StyleGAN-based portrait editing method is proposed, applied to a model including a decoder and an encoder. On the decoder side, the StyleGAN is reconstructed, expanding the receptive field of its first seven layers and changing its input from fixed 4x4 low-resolution features to variable-size high-resolution features. This allows it to support the generation and editing of non-aligned complete facial images without affecting its generation space and editing capabilities. On the encoder side, a corresponding face encoder is proposed, encoding the normal field of view facial image into style and content features as input to the reconstructed StyleGAN, implementing an end-to-end fully convolutional image transformation framework. Depending on the input and output data, training and model instances for various facial image editing tasks are provided, enabling applications such as face stylization, face super-resolution reconstruction, face attribute editing, and generating facial images from line drawings.

[0130] Part 1: Reconstructing the StyleGAN Decoder:

[0131] like Figure 2As shown, StyleGAN is reconstructed into a StyleGANEX decoder. Compared to StyleGAN, firstly, the fixed 4x4 resolution features in the first layer are replaced with a variable high-resolution feature of H*W, resulting in an output image resolution that changes from a fixed 1024*1024 to a variable 32H*32W. Secondly, for the first 7 layers of the network, upsampling is removed, and the convolutions in each layer are adjusted to corresponding dilated convolutions to match the receptive field. Specifically, the dilation rate of the first layer is 8, the dilation rate of layers 2-3 is 4, the dilation rate of layers 4-5 is 2, and the dilation rate of layers 6-7 is 1. Through this reconstruction, the input to the reconstructed StyleGANEX decoder becomes the style feature W. + (i.e., the style encoding mentioned above) and the content feature f of the first layer (i.e., the content feature mentioned above), wherein the style feature W + Consistent with StyleGAN, it controls the style of the face, while the content features f of the first layer encode the spatial information of the face in the image.

[0132] Part Two: Decoder for the Reconstructed StyleGAN:

[0133] 2.1 Network Model:

[0134] like Figure 4 As shown, a decoder for StyleGANEX was designed based on the original StyleGAN decoder. During model training, H×W content samples are input into the first feature layer module of the encoder. Compared to the original encoder, the style features W... + (including W1 to W) 18 The encoding of the input image remains unchanged. The intermediate feature layer in the first feature module outputs the intermediate features of the encoder (i.e., the first intermediate features mentioned above). Then, the intermediate features of the encoder are concatenated and passed through a convolutional layer (i.e., the transformation module mentioned above) to obtain the first layer content features f of the encoder, with a resolution of H / 8×W / 8. When forming an encoder-decoder image transformation framework with the decoder, optionally, the intermediate features of the encoder are output to the intermediate layer of the decoder (i.e., the first m feature layers mentioned above) through cross-layer connections, thereby better preserving the detailed information of the input image. The first layer content features f and style features W + After inputting into the StyleGANEX decoder, the decoder outputs a style image with a resolution of 4H×4W.

[0135] In some embodiments, the operation of having the encoder encode the image (i.e., the content sample mentioned above) into style code is denoted as E. W The operation of encoding the image (i.e., the style sample mentioned above) into the first-level content features (i.e., the content features mentioned above) is denoted as E. FLet l represent the output of the l feature layers of the encoder to the first l layers of the decoder (i.e., m above) through cross-layer connections. Let StyleGANEX be denoted as G, then the process of generating image x′ (i.e., style image above) based on content input x1 (i.e., the content sample above) and style input x2 (i.e., the style sample above) is shown in the following formula (2):

[0136] x′=G(E F (x1,l),E W (x2)) (2);

[0137] 2.2 Training Plan:

[0138] Based on x1 and x2, train x′ to approximate the real image x (i.e., the real sample mentioned above). The generalized training objective is shown in the following formula (3):

[0139] L = L rec (x′,x)+L other (x′) x′=G(E F (x1,l),E W (x2)) (3);

[0140] Among them, L rec It is the reconstruction loss, which measures the similarity between x′ and x. Commonly used metrics include L1 distance, L2 distance, perceptual distance, and identity similarity. other This is a loss function related to the task content of generating styled images, and its definition can vary depending on the task content. During testing, the corresponding face editing result x′ is obtained based on x1 and x2.

[0141] Below are examples of corresponding training schemes based on different editing tasks:

[0142] 2.2.1. Image Encoding and Decoding: This task encodes real human face images into f and W. + Then, it is decoded into the original face image. For the real face image x (i.e., the real sample mentioned above, which is also the content sample mentioned above), its training objective equation is shown in the following formula (4):

[0143] L = L rec (x′,x), x′=G(E) F (x,0),E W (x c (4);

[0144] When applying, by using W + After style editing, the image can be decoded into a face image with the corresponding style edit. c It is the aligned face region cropped from x.

[0145] 2.2.2. Face Super-Resolution Reconstruction: This task encodes low-resolution face images into f and W. + Then, it is decoded into a high-resolution face image. For the high-resolution face image x (i.e., the real sample mentioned above) and its corresponding low-resolution image y, the training objective equation is shown in the following formula (5):

[0146] L = L rec (x′,x)+L adv (x′,x), x′=G(E) F (y,7),E W (y c )) (5);

[0147] Among them, L adv It is the adversarial loss in generative adversarial networks, used to improve the realism of the output image. c (The style sample mentioned above) is an aligned face region cropped from y (the content sample mentioned above). When applied, any low-resolution image y can be input to obtain its high-resolution result x′.

[0148] 2.2.3. Generating Face Images from Line Artwork: This task generates corresponding color face images from line artwork describing the structure of a face. For a real face image x (i.e., the real sample mentioned above) and its corresponding line art image y, the training objective equation is shown in the following formula (6):

[0149]

[0150] Among them, y c and x c These are the aligned face regions cropped from the y and x axes, respectively. It is y c The first 7 elements of the extracted style features and x c The last 11 elements of the extracted style features are concatenated to form a new style feature. `l` is an adjustable parameter; the more detailed the line art, the larger `l` is to retain more of the line art structure, and vice versa, to optimize the line art structure as much as possible. In application, a face image is generated based on `y` and any style code.

[0151] 2.4. Video Face Attribute Editing: This task edits the faces in the input video to satisfy specified face attributes. For a real face image y, a face image x satisfying specified face attributes (changing hair color, changing expression, etc.), and a face attribute editing vector v, the training objective equation is shown in the following formula (7):

[0152] L = L rec (x′,x)+L adv (x′,x)+L tmp(x′); x′=G(E) F (y,13),E W (y c )+v) (7);

[0153] Among them, L tmp It is a loss function used to measure temporal continuity (i.e., the temporal continuity loss function mentioned above). In application, inputting a video frame y and an edit vector v will yield the corresponding edited video frame x′.

[0154] 2.2.5. Video Face Stylization: This task stylizes the faces in the input video. For a real face image y and a stylized face image x, the training objective equation is given by the following formula (8):

[0155] L = L rec (x′,x)+L adv (x′,x)+L tmp (x′), x′=G′(E) F (y,13),E W (y c )) (8);

[0156] Here, G′ is StyleGANEX, which loads the StyleGAN model finely tuned on the style dataset. When applied, inputting video frame y will yield the corresponding stylized video frame x′.

[0157] Existing StyleGAN-based technologies do not change the structure of StyleGAN itself, require face alignment of the input, and can only process 1024*1024 images. The solution provided in this application extends StyleGAN-based face image and video generation and editing technology to face images with a normal field of view, enabling the processing of misaligned face images with a larger field of view, thus improving the applicability and practicality of the technology. The solution provided in this application has the following effects:

[0158] Firstly, the embodiments of this application reconstruct the first 7 layers of StyleGAN, expanding its input to style features and variable resolution first-layer content features, thereby enabling the processing of non-aligned inputs with variable resolution.

[0159] Secondly, the embodiments of this application reconstruct the first 7 layers of StyleGAN without changing its style editing function. Therefore, it can realize editing tasks involving structural style, such as modifying facial expressions and perspectives.

[0160] Thirdly: Existing StyleGAN encoders only encode images into style features, which cannot be applied to the reconstructed StyleGAN; the encoder provided in this application provides additional first-layer content features with variable resolution, thereby enabling it to handle non-aligned inputs with variable resolution.

[0161] Fourthly, the encoder provided in this application embodiment encodes both content and style simultaneously and is trained end-to-end, thus enabling it to better complete the learning task.

[0162] The solutions provided in this application can be applied to the following scenarios:

[0163] 1. Face image restoration: This technology can convert low-resolution face images into high-resolution face images, which is suitable for the restoration of old photos, reconstructing the resolution of old photos, generating high-resolution photos, and using low-resolution images on high-definition display devices.

[0164] 2. Facial Image Editing: This technology can modify facial attributes, such as changing hair color and facial expressions. It also supports generating corresponding facial structures based on user-drawn line drawings. It is suitable for user-customized photo content (e.g., modifying facial attributes based on provided photos), and the design and generation of digital humans (e.g., designing and generating digital humans based on hand-drawn line drawings).

[0165] 3. Customize artistic avatars for users: This technology establishes a mapping relationship between human portraits and artistic portraits such as cartoons / comics / cartoons, allowing users to generate creative avatars in various styles based on their own photos. It also supports artistic face rendering in online meetings (by stylizing the video obtained from online meetings, artistic face rendering of online meetings can be achieved).

[0166] 4. Artistic Portrait Data Generation: This technology can be directly used to generate random artistic portraits. The generated image data can be used for the production of graphic products such as portrait design, animation posters and advertisements, and can also be used for subsequent deep learning and analysis as style examples for deep learning and analysis.

[0167] The solutions provided in this application can be applied to related products such as image restoration and reconstruction, artistic image design, live video streaming, video editing, and social platforms that display user avatars.

[0168] This application provides an image processing method applied to a model including an encoder and a first decoder, wherein the encoder includes a fully convolutional network, such as... Figure 5 As shown, the method includes the following steps S301 to S303:

[0169] Step S301: Obtain the content image to be processed with a first resolution, a style example matching the content image, and a preset second resolution, wherein there is a multiple relationship between the first resolution and the second resolution.

[0170] Here, the content image is the image that provides content during the generation of the style image in step S303. The style example is the image that provides style during the generation of the style image in step S303. The style image is an image generated according to different image generation tasks. Image generation tasks include, but are not limited to, face stylization, face super-resolution reconstruction, face attribute editing, and generating face images from line drawings.

[0171] The preset second resolution is the resolution of the style image generated in step S303, which can be a non-fixed resolution, i.e., any resolution. Since there is a multiple relationship between the first and second resolutions, the first resolution of the content image is also determined when the resolution of the generated style image is fixed. Furthermore, since the second resolution is non-fixed, the first resolution is also non-fixed, meaning the content image is an image at a non-fixed resolution.

[0172] Step S302: Obtain the content features of the content image and the first style code of the style sample through the encoder.

[0173] Step S303: Based on the content features and the first style encoding, generate an adjustable style image with the second resolution through the first decoder.

[0174] The method provided in this application is applied to a model including an encoder and a first decoder, wherein the encoder includes a fully convolutional network. First, based on the translation invariance of the fully convolutional network and the reusability of the convolutional kernels, the encoder can output content features at variable resolutions while outputting style codes. Second, the first decoder can generate an image of arbitrary resolution from content features reconstructed based on the StyleGAN decoder that can be input at non-fixed resolutions. In this way, it is possible to generate and edit aligned or misaligned images at non-fixed resolutions; that is, the input content image and the generated style image can be aligned or misaligned images at non-fixed resolutions, thereby solving the problem in related technologies that can only process fixed-size images and require face alignment.

[0175] In some embodiments, the encoder includes a first feature layer module and a transformation module. The implementation of step S302, "obtaining the content features of the content image and the first style encoding of the style sample through the encoder," includes the following steps S3021 to S3023:

[0176] Step S3021: Based on the preset second resolution and the first resolution of the content sample, determine at least one intermediate feature layer in the first feature layer module.

[0177] Step S3022: Based on the content image, obtain the first intermediate feature of the style image through the at least one intermediate feature layer.

[0178] Step S3023: Based on the first intermediate feature, the first intermediate feature is converted into a feature with the same number of channels in the first feature layer of the first decoder by the conversion module, so as to obtain the content features of the style image.

[0179] In this embodiment of the application, firstly, at least one intermediate feature layer for obtaining the first intermediate feature is determined in the first feature layer module; then, the first intermediate feature is obtained by using the features output by the at least one intermediate feature layer; finally, the first intermediate feature is converted into a feature with the same number of channels as the first feature layer of the first decoder by the conversion module, thereby obtaining the content features of the style image that the first decoder can process.

[0180] In some embodiments, the first decoder includes a second feature layer module, and the implementation of step S303, "generating an adjustable style image with the second resolution based on the content features and the first style encoding through the first decoder," includes the following steps S3031 to S3034:

[0181] Step S3031: In the first feature layer module, determine a set of target feature layers with the same resolution as the first m feature layers in the second feature layer module.

[0182] Step S3032: In the set of target feature layers, determine at least one target feature layer at each resolution.

[0183] Step S3033: Input the second intermediate feature output by the target feature layer in the at least one target feature layer into at least one feature layer with the same resolution in the second feature layer module.

[0184] Step S3034: Based on the content features, the first style encoding, and the second intermediate features input to the second feature layer module, generate an adjustable style image with the second resolution through the first decoder.

[0185] In this embodiment of the application, by inputting the second intermediate feature output by the target feature layer in the first feature layer module into at least one feature layer with the same resolution in the second feature layer module, more structural information is provided to the style image generated by the first decoder, thereby making the generated style image include more detailed information.

[0186] In some embodiments, the implementation of step S3031, "determining in the first feature layer module a set of target feature layers with the same resolution as the first m feature layers in the second feature layer module", includes the following steps S401 and S402:

[0187] Step S401: Determine the value of m based on the task content of generating the style image.

[0188] Step S402: In the first feature layer module, determine a set of target feature layers with the same resolution as the first m feature layers in the second feature layer module.

[0189] In this embodiment, the value of m in the first m feature layers is determined according to the task content of generating the style image, so as to set the number of feature layers in the input second feature layer module for different task content, so that the generated style image can better display the structural information contained in the content image.

[0190] In some embodiments, when the task of generating the style image includes image encoding / decoding or face super-resolution reconstruction, the implementation of "obtaining the content image to be processed with a first resolution and the style sample matching the content image" in step S301 may include the following steps S501 to S503:

[0191] Step S501: Obtain the first image to be processed with a first resolution.

[0192] Here, when the task of generating the style image includes image encoding and decoding, the first image can be the original image that needs to be encoded and decoded; when the task of generating the style image includes face super-resolution reconstruction, the first image can be a low-resolution face image, such as an old photo that needs to be restored.

[0193] Step S502: Based on the first image, crop to obtain a second image with a preset size and alignment.

[0194] Here, the implementation of step S502 may include: performing face detection on the first image to obtain a face detection box in the first image; cropping out an area of ​​a preset size with the center of the face detection box as the center, which is the second image.

[0195] Step S503: Use the first image as the content image and the second image as the style example.

[0196] It should be noted that in some embodiments, the aligned image may not be used as a style example, that is, the first image may be used directly as a style example. This application does not limit whether the style example is aligned.

[0197] This application provides a method for obtaining content images and style samples when the task of generating style images includes image encoding / decoding or face super-resolution reconstruction. In this case, style samples can be obtained by cropping the content image.

[0198] In some embodiments, where the task of generating the style image includes image encoding and decoding, after step S302 "obtaining the first style code of the style sample through the encoder", the method further includes the following step S601:

[0199] Step S601: Edit the first style code to obtain the edited second style code.

[0200] Here, the second style code is the style code corresponding to the facial attribute editing required as defined in the task content. For example, if it is necessary to enlarge the eyes in the original image, then the second style code is the style code corresponding to enlarging the eyes. The implementation of step S601 means that during image encoding and decoding, the style code of the first image (i.e., the original image) can be edited to obtain an image with edited facial attributes.

[0201] Correspondingly, the implementation of step S303, "based on the content features and the first style encoding, generating an adjustable style image with the second resolution through the first decoder," may include:

[0202] Based on the content features and the second style encoding, an adjustable style image with the second resolution is generated by the first decoder.

[0203] This application provides a method for generating a style image with edited facial attributes when the task of generating a style image includes image encoding and decoding. Specifically, a first style code is first edited to a second style code corresponding to the facial attribute edit to be performed. Then, content features and the second style code are used to generate the style image with edited facial attributes.

[0204] In some embodiments, when the task of generating the style image includes generating a face image based on a line drawing, the implementation of "obtaining a content image to be processed with a first resolution and a style sample matching the content image" in step S301 may include the following steps S701 to S703:

[0205] Step S701: Obtain the line art image to be processed and the third image with the style to be generated.

[0206] Here, the line art image refers to an image composed of lines without color. The third image is the image containing the style to be generated. In practice, the third image can be an image with any style.

[0207] Step S702: Based on the line drawing image and the third image, respectively, crop to obtain a fourth image and a fifth image with a preset size and alignment.

[0208] Step S703: Use the line drawing image as the content image, and use both the fourth image and the fifth image as style examples.

[0209] Here, since the fourth image comes from the line drawing, it can provide more structural information; while the fifth image comes from the third image with the style to be generated, it can provide more color information. Therefore, both the fourth and fifth images can be used as style examples. By combining the advantages of the fourth and fifth images, a style code can be obtained. For example, the first seven layers of the style code obtained from the fourth image and the last 11 layers of the style code obtained from the fifth image can be concatenated to obtain the style code, thus making the obtained style code contain more structural and color information.

[0210] Correspondingly, the implementation of "obtaining the first style code of the style sample through the encoder" in step S302 may include the following steps S801 to S803:

[0211] Step S801: Input the fourth image into the encoder to obtain the third style code used to influence the style image structure.

[0212] Here, the third style coding can be the style coding of the first seven layers output after the fourth image input encoder.

[0213] Step S802: Input the fifth image into the encoder to obtain the fourth style code used to influence the color of the style image.

[0214] Here, the fourth style coding can be the style coding of the output of the last eleven layers after the fifth image input encoder.

[0215] Step S803: Concatenate the third style code and the fourth style code to obtain the first style code of the style example.

[0216] In this embodiment of the application, the style code is obtained by concatenating the first seven layers of the style code obtained from the fourth image with the last 11 layers of the style code obtained from the fifth image. This results in the style code containing more structural and color information, which in turn enables the final generated style image to more accurately reflect the structural features of the line drawing image and the style features of the style example.

[0217] In some embodiments, when the task of generating the style image includes video face attribute editing or video face stylization, the implementation of "obtaining the content image to be processed with a first resolution and the style sample matching the content image" in step S301 may include the following steps S901 to S903:

[0218] Step S901: Obtain the video to be processed.

[0219] Here, the video to be processed refers to the video that needs to have its facial attributes edited or its face stylized, such as videos in products like short video live streaming, virtual anchors, and online meetings.

[0220] Step S902: For each frame of the video, crop to obtain a sixth image with a preset size and alignment.

[0221] Step S903: Take each frame of the video as the content image, and take the sixth image corresponding to each frame as the style example.

[0222] In this embodiment of the application, by using each frame of the video as a content image and each cropped frame as a style example, and inputting them into the model provided in this embodiment of the application, it is possible to edit facial attributes or stylize facial features in the video.

[0223] In some embodiments, where the task of generating the style image includes video face attribute editing, the method further includes the following step: obtaining an edit vector for editing the face attributes in the video.

[0224] Here, the edit vector refers to the vector used to edit facial attributes. Editing facial attributes can include things like enlarging the eyes or removing moles.

[0225] Correspondingly, after step S302 "obtaining the first style code of the style sample through the encoder", the following steps are also included:

[0226] The fifth style code is obtained by adding the first style code and the edit vector.

[0227] Correspondingly, the implementation of step S303, "based on the content features and the first style encoding, generate an adjustable style image with the second resolution through the first decoder", may include the following steps: based on the content features and the fifth style encoding, generate an adjustable style image with the second resolution through the first decoder.

[0228] This application provides a method for generating style images when the task of generating style images includes video face attribute editing. Since the task of generating style images involves video face attribute editing, it is necessary to add the first style code and the edit vector to obtain the fifth style code, and then use the content features and the fifth style code to generate the style image.

[0229] In some embodiments, when the task of generating the style image includes video face stylization, the implementation of step S303 "generating an adjustable style image with the second resolution through the first decoder based on the content features and the first style encoding" may include: generating an adjustable style image with the second resolution through a second decoder based on the content features and the first style encoding, wherein the second decoder is a decoder obtained by fine-tuning the first decoder using a set of style samples.

[0230] This application provides a method for generating style images when the task of generating style images includes face stylization. Since the task of generating style images is video face stylization, it is necessary to first fine-tune the first decoder to obtain a second decoder capable of generating style images, and then use the style images generated by the second decoder.

[0231] Based on the foregoing embodiments, this application provides a model training device, which includes various units and modules included in each unit, and can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0232] Figure 6 This is a schematic diagram of the composition structure of a model training device provided in an embodiment of this application. The model includes an encoder and a first decoder. The encoder includes a fully convolutional network, such as... Figure 6As shown, the model training device 600 includes: a first acquisition module 610, a second acquisition module 620, a first generation module 630, and a training module 640, wherein:

[0233] The first acquisition module 610 is used to acquire a content sample set, a style sample set, and a real sample set, wherein the content samples in the content sample set are images with a first resolution, and there is a multiple relationship between the first resolution and a preset second resolution;

[0234] The second acquisition module 620 is used to acquire the content features of each content sample and the first style code of the style sample corresponding to each content sample in the style sample set through the encoder.

[0235] The first generation module 630 is used to generate an adjustable style image with the second resolution based on the content features of each content sample and the first style code corresponding to the content features, through the first decoder, and obtain a style image set including the style image.

[0236] The training module 640 is used to iteratively train the model based on each style image in the style image set and the real samples corresponding to each style image in the real sample set until the convergence condition is met.

[0237] In some embodiments, the first decoder is a decoder reconstructed based on the StyleGAN decoder, which processes content features at a first non-fixed resolution in an equivalent manner to how the StyleGAN decoder processes content features at a fixed resolution, in order to generate a style image at a second non-fixed resolution.

[0238] In some embodiments, the encoder includes a first feature layer module and a conversion module, and the second acquisition module includes: a first determining submodule, configured to determine at least one intermediate feature layer in the first feature layer module for each content sample based on the preset second resolution and the first resolution of the content sample; a first acquisition submodule, configured to acquire a first intermediate feature of a style image corresponding to the content sample through the at least one intermediate feature layer based on each content sample; and a first conversion submodule, configured to convert the first intermediate feature into a feature with the same number of channels as the first feature layer of the first decoder through the conversion module, based on the first intermediate feature, to acquire the content feature of each content sample.

[0239] In some embodiments, the first decoder includes a second feature layer module, and the first generation module includes: a second determining submodule, configured to determine a set of target feature layers with the same resolution as the first m feature layers in the second feature layer module; a third determining submodule, configured to determine at least one target feature layer at each resolution in the set of target feature layers; a first input submodule, configured to input a second intermediate feature output by the target feature layer in the at least one target feature layer into at least one feature layer with the same resolution in the second feature layer module; and a first generation submodule, configured to generate an adjustable style image with the second resolution based on the content features of each content sample, the style encoding corresponding to the content features, and the second intermediate feature input to the second feature layer module, through the first decoder.

[0240] In some embodiments, the second determining submodule includes: a first determining unit, configured to determine the value of m based on the task content of generating the style image; and a second determining unit, configured to determine a set of target feature layers in the first feature layer module that has the same resolution as the first m feature layers in the second feature layer module.

[0241] In some embodiments, the training module includes: a fourth determining submodule, configured to determine a loss function for training the model based on the task content of generating the style images; a fifth determining submodule, configured to determine a loss value corresponding to the loss function based on each style image in the generated style image set and the real sample corresponding to each style image; and a completion submodule, configured to complete the training of the model when the loss value satisfies the convergence condition.

[0242] In some embodiments, when the task of generating the style image includes image encoding / decoding or generating a face image from a line drawing, the loss function includes a reconstruction loss, wherein the reconstruction loss is used to characterize the similarity between the generated style image and the corresponding real sample; and / or, when the task of generating the style image includes face super-resolution reconstruction, the loss function includes the reconstruction loss and an adversarial loss, wherein the adversarial loss is the adversarial loss of the first decoder; and / or, when the task of generating the style image is video face attribute editing or video face stylization, the loss function includes the reconstruction loss, the adversarial loss, and a temporal continuity loss, wherein the temporal continuity loss is used to characterize the consistency of semantically identical pixels in the style images generated from different image frames in the video.

[0243] This application also provides an image processing apparatus applied to a model including an encoder and a first decoder. The encoder includes a fully convolutional network. The image processing apparatus includes: a third acquisition module, a fourth acquisition module, and a second generation module, wherein:

[0244] The third acquisition module is used to acquire a content image to be processed with a first resolution, a style example matching the content image, and a preset second resolution, wherein there is a multiple relationship between the first resolution and the second resolution;

[0245] The fourth acquisition module is used to acquire the content features of the content image and the first style code of the style sample through the encoder;

[0246] The second generation module is used to generate an adjustable style image with the second resolution based on the content features and the first style encoding through the first decoder.

[0247] In some embodiments, the encoder includes a first feature layer module and a conversion module, and the fourth acquisition module includes: a sixth determining submodule, configured to determine at least one intermediate feature layer in the first feature layer module based on the preset second resolution and the first resolution of the content sample; a second acquisition submodule, configured to acquire a first intermediate feature of the style image based on the content image through the at least one intermediate feature layer; and a second conversion submodule, configured to convert the first intermediate feature into a feature with the same number of channels as the first feature layer of the first decoder, based on the first intermediate feature, through the conversion module, to acquire the content feature of the style image.

[0248] In some embodiments, the first decoder includes a second feature layer module, and the second generation module includes: a seventh determining submodule, configured to determine a set of target feature layers with the same resolution as the first m feature layers in the second feature layer module; an eighth determining submodule, configured to determine at least one target feature layer at each resolution in the set of target feature layers; a second input submodule, configured to input a second intermediate feature output by the target feature layer in the at least one target feature layer into at least one feature layer with the same resolution in the second feature layer module; and a second generation submodule, configured to generate an adjustable style image with the second resolution by means of the first decoder based on the content feature, the first style encoding, and the second intermediate feature input to the second feature layer module.

[0249] In some embodiments, the seventh determining submodule includes: a third determining unit, configured to determine the value of m based on the task content of generating the style image; and a fourth determining unit, configured to determine a set of target feature layers in the first feature layer module that has the same resolution as the first m feature layers in the second feature layer module.

[0250] In some embodiments, when the task of generating the style image includes image encoding / decoding or face super-resolution reconstruction, the third acquisition module includes: a third acquisition submodule, used to acquire a first image to be processed with a first resolution; a first cropping submodule, used to crop and acquire an aligned second image with a preset size based on the first image; and an eighth determination submodule, used to use the first image as the content image and the second image as the style example.

[0251] In some embodiments, where the task of generating the style image includes image encoding and decoding, after obtaining the first style code of the style sample through the encoder, the apparatus further includes: an editing module for editing the first style code to obtain an edited second style code; and a second generation module for generating an adjustable style image with the second resolution through the first decoder based on the content features and the second style code.

[0252] In some embodiments, when the task of generating the style image includes generating a face image based on a line drawing, the third acquisition module includes: a fourth acquisition submodule, used to acquire a line drawing image to be processed and a third image with a style to be generated; a second cropping submodule, used to crop and acquire aligned fourth and fifth images with preset sizes based on the line drawing image and the third image respectively; a ninth determination submodule, used to use the line drawing image as the content image and both the fourth and fifth images as style examples; correspondingly, the fourth acquisition module includes: a fifth acquisition submodule, used to input the fourth image into the encoder to acquire a third style code for influencing the structure of the style image; a sixth acquisition submodule, used to input the fifth image into the encoder to acquire a fourth style code for influencing the color of the style image; and a concatenation submodule, used to concatenate the third style code and the fourth style code to obtain a first style code for the style example.

[0253] In some embodiments, when the task of generating the style image includes video face attribute editing or video face stylization, the third acquisition module includes: a seventh acquisition submodule, used to acquire the video to be processed; a third cropping submodule, used to crop and acquire a sixth image with a preset size for each frame of the video; and a tenth determination submodule, used to take each frame of the video as the content image and the sixth image corresponding to each frame as the style example.

[0254] In some embodiments, when the task of generating the style image includes video face attribute editing, the apparatus further includes: a fifth acquisition module, configured to acquire an edit vector for editing face attributes in the video; correspondingly, after acquiring the first style code of the style example through the encoder, the apparatus further includes: an addition module, configured to add the first style code and the edit vector to obtain a fifth style code; the second generation module is further configured to generate an adjustable style image with the second resolution through the first decoder based on the content features and the fifth style code.

[0255] In some embodiments, where the task of generating the style image includes video face stylization, the second generation module is further configured to generate an adjustable style image with the second resolution based on the content features and the first style encoding by a second decoder, wherein the second decoder is a decoder obtained by fine-tuning the first decoder using a set of style samples.

[0256] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0257] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0258] It should be noted that, in the embodiments of this application, if the above-described model training and image processing methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0259] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0260] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0261] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0262] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0263] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0264] It should be noted that, Figure 7 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 7 As shown, the hardware entity of the computer device 700 includes: a processor 701, a communication interface 702, and a memory 703, wherein:

[0265] Processor 701 typically controls the overall operation of computer device 700.

[0266] Communication interface 702 enables computer devices to communicate with other terminals or servers over a network.

[0267] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 701 and various modules in the computer device 700. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 701, the communication interface 702, and the memory 703 can be performed via bus 704.

[0268] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0269] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0270] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0271] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0272] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0273] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory, magnetic disks, or optical disks.

[0274] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0275] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, The model includes an encoder and a first decoder, the encoder including a fully convolutional network, and the method includes: Obtain a content sample set, a style sample set, and a real sample set, wherein the content samples in the content sample set are images with a first resolution, and there is a multiple relationship between the first resolution and a preset second resolution; The encoder obtains the content features of each content sample and the first style code of the style sample corresponding to each content sample in the style sample set. Based on the content features of each content sample and the first style code corresponding to the content features, an adjustable style image with the second resolution is generated through the first decoder, and a style image set including the style image is obtained; the first decoder is a decoder reconstructed based on the StyleGAN decoder, and uses an equivalent method to the StyleGAN decoder to process the content features under the first non-fixed resolution, so as to generate the style image of the second non-fixed resolution. Based on each style image in the style image set and the real samples corresponding to each style image in the real sample set, the model is iteratively trained until the convergence condition is met. The reconstruction of the StyleGAN decoder includes: The fixed 4x4 resolution features of the first layer of the StyleGAN decoder are replaced with variable H*W high resolution features. The upsampling operation is removed from the first 7 layers of the StyleGAN decoder, and the convolutions of each layer are adjusted to corresponding dilated convolutions to match the receptive field; wherein, the dilation rate of the first layer is 8, the dilation rate of the second to third layers is 4, the dilation rate of the fourth to fifth layers is 2, and the dilation rate of the sixth to seventh layers is 1.

2. The training method according to claim 1, characterized in that, The encoder includes a first feature layer module and a transformation module. The step of obtaining content features for each content sample through the encoder includes: For each content sample, at least one intermediate feature layer is determined in the first feature layer module based on the preset second resolution and the first resolution of the content sample; Based on each content sample, a first intermediate feature of the style image corresponding to the content sample is obtained through the at least one intermediate feature layer; Based on the first intermediate feature, the conversion module transforms the first intermediate feature into a feature with the same number of channels in the first feature layer of the first decoder, so as to obtain the content feature of each content sample.

3. The training method according to claim 2, characterized in that, The first decoder includes a second feature layer module. Based on the content features of each content sample and a first style code corresponding to those content features, the first decoder generates an adjustable style image with the second resolution, including: In the first feature layer module, a set of target feature layers with the same resolution as the first m feature layers in the second feature layer module is determined; In the set of target feature layers, at least one target feature layer is determined for each resolution; The second intermediate feature output by the target feature layer in the at least one target feature layer is input into at least one feature layer with the same resolution in the second feature layer module; Based on the content features of each content sample, the style encoding corresponding to the content features, and the second intermediate features input into the second feature layer module, an adjustable style image with the second resolution is generated through the first decoder.

4. The training method according to claim 3, characterized in that, The step of determining a set of target feature layers in the first feature layer module that has the same resolution as the first m feature layers in the second feature layer module includes: The value of m is determined based on the task content of generating the style image; In the first feature layer module, a set of target feature layers with the same resolution as the first m feature layers in the second feature layer module is determined.

5. The training method according to claim 1, characterized in that, The step of iteratively training the model based on each style image in the style image set and the real samples corresponding to each style image in the real sample set until the convergence condition is met includes: Based on the task content of generating the style image, determine the loss function used to train the model; Based on each style image in the generated style image set and the real sample corresponding to each style image, the loss value corresponding to the loss function is determined. Training of the model is completed when the loss value meets the convergence condition.

6. The training method according to claim 5, characterized in that, When the task of generating the style image includes image encoding / decoding or generating a face image from a line drawing, the loss function includes a reconstruction loss, wherein the reconstruction loss is used to characterize the similarity between the generated style image and the corresponding real sample; And / or, when the task content for generating the style image includes face super-resolution reconstruction, the loss function includes the reconstruction loss and the adversarial loss, wherein the adversarial loss is the adversarial loss of the first decoder; And / or, when the task of generating the style image is video face attribute editing or video face stylization, the loss function includes the reconstruction loss, the adversarial loss, and the temporal continuity loss, wherein the temporal continuity loss is used to characterize the consistency of semantically identical pixels in the style images generated from different image frames in the video.

7. An image processing method, characterized in that, The method is applied to a model including an encoder and a first decoder, the encoder including a fully convolutional network, and the method includes: The process involves acquiring a content image with a first resolution to be processed, a style example matching the content image, and a preset second resolution, wherein there is a multiple relationship between the first resolution and the second resolution. The encoder is used to obtain the content features of the content image and the first style code of the style sample; Based on the content features and the first style encoding, an adjustable style image with the second resolution is generated by the first decoder; the first decoder is a decoder reconstructed based on the StyleGAN decoder, and uses an equivalent method to the StyleGAN decoder to process the content features at the first non-fixed resolution in order to generate a style image at the second non-fixed resolution. The reconstruction of the StyleGAN decoder includes: replacing the fixed 4*4 resolution features of the first layer of the StyleGAN decoder with variable H*W high-resolution features; removing upsampling operations from the first 7 layers of the StyleGAN decoder and adjusting the convolution of each layer to a corresponding dilated convolution to match the receptive field; wherein the dilation rate of the first layer is 8, the dilation rate of the second to third layers is 4, the dilation rate of the fourth to fifth layers is 2, and the dilation rate of the sixth to seventh layers is 1.

8. The method according to claim 7, characterized in that, When the task of generating the style image includes image encoding / decoding or face super-resolution reconstruction, the step of obtaining the content image to be processed with a first resolution and the style sample matching the content image includes: Acquire a first image with a first resolution to be processed; Based on the first image, a second image with a preset size and alignment is obtained by cropping; The first image is used as the content image, and the second image is used as the style example.

9. The method according to claim 7, characterized in that, When the task of generating the style image includes generating a face image based on a line drawing, the step of obtaining the content image to be processed with a first resolution and the style sample matching the content image includes: Obtain the line art image to be processed and a third image with the style to be generated; Based on the line drawing image and the third image respectively, crop to obtain an aligned fourth image and a fifth image with a preset size; The line drawing image is used as the content image, and the fourth and fifth images are both used as style examples; Correspondingly, obtaining the first style code of the style sample through the encoder includes: The fourth image is input into the encoder to obtain a third style code that influences the structure of the style image. The fifth image is input into the encoder to obtain a fourth style code that affects the color of the style image; By concatenating the third style code and the fourth style code, the first style code of the style example is obtained.

10. The method according to claim 7, characterized in that, When the task of generating the styled image includes video face attribute editing, the method further includes: Obtain the edit vector for editing facial attributes in the video; Correspondingly, after obtaining the first style code of the style sample through the encoder, the method further includes: The fifth style code is obtained by adding the first style code and the edit vector. The step of generating an adjustable style image with the second resolution using the first decoder based on the content features and the first style encoding includes: Based on the content features and the fifth style encoding, an adjustable style image with the second resolution is generated by the first decoder.

11. A model training device, characterized in that, The model includes an encoder and a first decoder, the encoder including a fully convolutional network, and the model training device including: The first acquisition module is used to acquire a content sample set, a style sample set, and a real sample set, wherein the content samples in the content sample set are images with a first resolution, and there is a multiple relationship between the first resolution and a preset second resolution; The second acquisition module is used to acquire the content features of each content sample and the first style code of the style sample corresponding to each content sample in the style sample set through the encoder. The first generation module is used to generate an adjustable style image with a second resolution based on the content features of each content sample and the first style code corresponding to the content features, through the first decoder, and obtain a style image set including the style image; the first decoder is a decoder reconstructed based on the StyleGAN decoder, and uses an equivalent method to the StyleGAN decoder to process the content features under the first non-fixed resolution, so as to generate the style image of the second non-fixed resolution. The training module is used to iteratively train the model based on each style image in the style image set and the real samples corresponding to each style image in the real sample set until the convergence condition is met. The reconstruction of the StyleGAN decoder includes: replacing the fixed 4*4 resolution features of the first layer of the StyleGAN decoder with variable H*W high-resolution features; removing upsampling operations from the first 7 layers of the StyleGAN decoder and adjusting the convolution of each layer to a corresponding dilated convolution to match the receptive field; wherein the dilation rate of the first layer is 8, the dilation rate of the second to third layers is 4, the dilation rate of the fourth to fifth layers is 2, and the dilation rate of the sixth to seventh layers is 1.

12. An image processing apparatus, characterized in that, The image processing apparatus is applied to a model including an encoder and a first decoder, the encoder including a fully convolutional network, and the image processing apparatus comprising: The third acquisition module is used to acquire a content image to be processed with a first resolution, a style example matching the content image, and a preset second resolution, wherein there is a multiple relationship between the first resolution and the second resolution; The fourth acquisition module is used to acquire the content features of the content image and the first style code of the style sample through the encoder; The second generation module is used to generate an adjustable style image with the second resolution based on the content features and the first style encoding through the first decoder; the first decoder is a decoder reconstructed based on the StyleGAN decoder, and uses an equivalent method to the StyleGAN decoder to process the content features under the first non-fixed resolution in order to generate the style image of the second non-fixed resolution. The reconstruction of the StyleGAN decoder includes: replacing the fixed 4*4 resolution features of the first layer of the StyleGAN decoder with variable H*W high-resolution features; removing upsampling operations from the first 7 layers of the StyleGAN decoder and adjusting the convolution of each layer to a corresponding dilated convolution to match the receptive field; wherein the dilation rate of the first layer is 8, the dilation rate of the second to third layers is 4, the dilation rate of the fourth to fifth layers is 2, and the dilation rate of the sixth to seventh layers is 1.

13. A computer device, characterized in that, include: Memory and processor The memory stores computer programs that can run on the processor. When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6, or implements the steps of the method according to any one of claims 7 to 10.

14. A computer storage medium, characterized in that, The computer storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the method according to any one of claims 1 to 6, or to implement the steps of the method according to any one of claims 7 to 10.

Images