Image generation model training method and device, computer device, and storage medium

Abstract

The application discloses a training method and device of an image generation model, a computer device and a storage medium, and belongs to the technical field of image processing. The application obtains multiple style features belonging to different styles through an image generation model, determines a first error and a second error based on the distribution of the multiple style features in space, and applies the first error and the second error for model training. The application of the first error can constrain the multiple style features obtained by the image generation model in a compact space, so that the style features belonging to different styles can be smoothly transitioned. The application of the second error can maintain the separability between the style features belonging to different styles. That is, the above training method can maintain the distinguishability and smooth transition between the multiple style features, so that the image generation model can ensure good transition between an original image and a newly generated image when converting the style of the image, and improve the authenticity and image quality of the newly generated image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method, apparatus, computer equipment, and storage medium for an image generation model. Background Art

[0002] In the field of image processing, computers can apply neural network models to transform the style of an image and generate a new image. For example, using image generation models, computers can transform the style of a face image to perform gender conversion, makeup changes, and more.

[0003] However, when applying image generative models to transfer graphic styles and generate new images, the difficulty in achieving a smooth transition between different image styles leads to low fidelity and poor image quality. Therefore, how to train image generative models, improve their performance, and enhance the fidelity of generated images is an important research direction. Summary of the Invention

[0004] The present invention provides a method, apparatus, computer device, and storage medium for training an image generation model, which can improve the model performance of the image generation model when performing image style conversion tasks, and improve the realism and image quality of images generated by the image generation model. The technical solution is as follows:

[0005] In one aspect, a method for training an image generation model is provided, the method comprising:

[0006] Obtaining an image generation model to be trained, the image generation model is used to convert any style information into corresponding style features, and convert the style of any image based on the style features, the style information being information used to describe the corresponding style;

[0007] Obtain at least two sets of style information, where the style information in the same set corresponds to the same style;

[0008] Each style information is converted into corresponding style features through the image generation model;

[0009] Obtaining a first error and a second error based on the converted multiple style features, distribution information of the multiple style features, and target distribution information, wherein the target distribution information is used to define a spatial distribution area of ​​the multiple style features, the first error is used to indicate an error between the distribution information of the multiple style features and the target distribution information, and the second error is used to indicate a difference in spatial distribution of style features corresponding to different styles;

[0010] Based on the first error and the second error, parameters of the image generation model are adjusted.

[0011] In one aspect, a training apparatus for an image generation model is provided, the apparatus comprising:

[0012] A model acquisition module is used to obtain an image generation model to be trained. The image generation model is used to convert any style information into corresponding style features, and convert the style of any image based on the style features. The style information is information used to describe the corresponding style.

[0013] An information acquisition module, configured to acquire at least two sets of style information, wherein the style information in the same set corresponds to the same style;

[0014] An information conversion module, configured to convert each style information into a corresponding style feature through the image generation model;

[0015] an error acquisition module, configured to acquire a first error and a second error based on a plurality of style features obtained by conversion, distribution information of the plurality of style features, and target distribution information, wherein the target distribution information is used to define a spatial distribution area of ​​the plurality of style features, the first error is used to indicate an error between the distribution information of the plurality of style features and the target distribution information, and the second error is used to indicate a difference in spatial distribution of style features corresponding to different styles;

[0016] A parameter adjustment module is used to adjust the parameters of the image generation model based on the first error and the second error.

[0017] In one possible implementation, the information conversion module is configured to perform any of the following:

[0018] In response to the style information including an image, extracting style features of the image through a feature extraction network in the image generation model;

[0019] In response to the style information including a style label, the style label is mapped to a corresponding style feature through a feature mapping network in the image generation model.

[0020] In one possible implementation, the error acquisition module is used to

[0021] Obtaining the KL divergence between the distribution information of the plurality of style features and the target distribution information;

[0022] determining the first error based on the KL divergence, where the first error is positively correlated with the KL divergence;

[0023] Obtaining a first distance between any two style features belonging to the same style and a second distance between any two style features belonging to different styles;

[0024] The second error is determined based on a difference between the first distance and the second distance corresponding to the same style feature.

[0025] In one possible implementation, the device further includes:

[0026] The image acquisition module is used to convert the style of the initial image to be processed based on any style feature through the generation network in the image generation model to obtain a target image, where the style of the target image is consistent with the style indicated by any style feature.

[0027] In one possible implementation, the device further includes:

[0028] an error determination module, configured to determine a third error based on a similarity between the target image and the initial image;

[0029] The parameter adjustment module is used to adjust the parameters of the image generation model based on the first error, the second error and the third error.

[0030] In one possible implementation, the error determination module is configured to:

[0031] Performing feature extraction of different scales on the target image and the initial image through at least two scale feature extraction layers, respectively, to obtain at least two first features corresponding to the target image and at least two second features corresponding to the initial image;

[0032] respectively obtaining a third distance between the first feature and the second feature of the same scale;

[0033] The third error is determined based on the acquired at least two third distances.

[0034] In one possible implementation, the device further includes:

[0035] a weighted processing module, configured to perform weighted processing on the first error, the second error, and the third error to obtain a fourth error;

[0036] The parameter adjustment module is used to adjust the parameters of the image generation model based on the fourth error.

[0037] On the one hand, a computer device is provided, which includes one or more processors and one or more memories, wherein the one or more memories store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors to implement the operations performed by the training method of the image generation model.

[0038] On the one hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed by the training method of the image generation model.

[0039] In one aspect, a computer program product is provided, comprising at least one computer program stored in a computer-readable storage medium. A processor of a computer device reads the at least one computer program from the computer-readable storage medium and executes the at least one computer program, causing the computer device to implement the operations performed by the method for training an image generation model.

[0040] The technical solution provided by the embodiment of the present application obtains multiple style features belonging to different styles through an image generation model, determines a first error and a second error based on the spatial distribution of the multiple style features, and applies the first error and the second error to train the image generation model, wherein the application of the first error can constrain the spatial distribution of the style features obtained by the image generation model, so that the distribution of the multiple style features is within a compact space, thereby enabling a smooth transition between the style features belonging to different styles, and the application of the second error can enable the style features belonging to different styles to be separable, that is, through this model training method, the style features belonging to different styles can maintain both distinguishability and smooth transition, so that when the image generation model performs a conversion of the initial image with a large style span, it ensures a good transition between the input image and the newly generated image, thereby improving the authenticity and image quality of the newly generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 Schematic diagram of an implementation environment of a training method for an image generation model provided in an embodiment of the present application;

[0043] Figure 2 This is a flowchart of a method for training an image generation model provided in an embodiment of the present application;

[0044] Figure 3 is a schematic diagram of an image generation model provided in an embodiment of the present application;

[0045] Figure 4This is a flowchart of a method for training an image generation model provided in an embodiment of the present application;

[0046] Figure 5 This is a schematic diagram of the distribution of style features provided in an embodiment of the application;

[0047] Figure 6 This is a performance comparison diagram of an image generation model trained using different model training methods provided in an embodiment of the present application;

[0048] Figure 7 Schematic diagram of a training device for an image generation model provided in an embodiment of the present application;

[0049] Figure 8 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0050] Figure 9 This is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of this application more clear, the following will further describe the embodiments of this application in detail with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0052] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on the quantity and execution order.

[0053] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to perceive, reason, and make decisions. AI technology is an interdisciplinary discipline covering a wide range of fields, encompassing both hardware and software technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning. The embodiments of this application relate to computer vision and deep learning within AI technology.

[0054] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying, tracking, and measuring objects, and further processing them to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0055] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0056] To facilitate understanding of the technical process of the embodiments of the present application, some terms involved in the embodiments of the present application are explained below:

[0057] Image Domain: refers to a collection of images with the same image style. For example, female face images belong to the same image domain, male face images belong to the same image domain, and sketch-style images belong to the same image domain.

[0058] Unsupervised Image-to-image Translation: refers to the process of training a neural network model to perform style conversion on the input image and generate a new image without the real target result as supervision data.

[0059] Latent Style Space: refers to representing the style of an image in a lower-dimensional vector space.

[0060] Style Code: refers to the vector corresponding to the image in the style latent space. In the embodiment of the present application, the style features of the image are represented in the form of a style vector.

[0061] Figure 1 This is a schematic diagram of the implementation environment of a training method for an image generation model provided in an embodiment of the present application, see Figure 1 , the implementation environment includes a terminal 110 and a server 140 .

[0062] Among them, the terminal 110 installs and runs an application that supports image style transformation, for example, the application is an image processing application, an image shooting application, a social application, etc., and the embodiments of the present application do not limit this. Exemplarily, in the image processing application, the user inputs an initial image to be processed and then inputs a reference image with a target style. The style of the initial image is different from that of the reference image. The image processing application can perform style transformation on the initial image based on the image style of the reference image to generate a new image. The newly generated image has the image style of the reference image, that is, the newly generated image also has the target style. Optionally, the terminal 110 is a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., and the embodiments of the present application do not limit the device type of the terminal 110. Exemplarily, the terminal 110 is a terminal used by the user, and a user account is logged in to the application running in the terminal 110. The terminal 110 generally refers to one of multiple terminals. This embodiment only uses the terminal 110 as an example.

[0063] In one possible implementation, the server 140 is at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 140 is used to provide background services for applications for image style conversion. Optionally, the server 140 undertakes the main image processing work, and the terminal 110 undertakes the secondary image processing work; or, the server 140 undertakes the secondary image processing work, and the terminal 110 undertakes the main image processing work; or, the server 140 or the terminal 110 can each undertake the image processing work independently. Optionally, the server 140 includes: an access server, an image processing server, and a database. The access server is used to provide access services for the terminal 110. The image processing server is used to provide background services related to image style conversion. Exemplarily, the image processing server can be equipped with an image processor (Graphics Processing Unit, GPU) to support multi-threaded parallel computing of the image processor. Exemplarily, there are one or more image processing servers. When there are multiple image processing servers, there are at least two image processing servers for providing different services, and / or there are at least two image processing servers for providing the same service, such as providing the same service in a load balancing manner, which is not limited in this embodiment of the present application. An image generation model can be provided in the image processing server. During the model training and application process, the image processing server can be equipped with an image processor and support parallel computing of the image processor. Exemplarily, the above-mentioned server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The embodiment of the present application does not limit the number of servers and device types.

[0064] The embodiment of the present application provides a training method for an image generation model. The image generation model involved in the embodiment of the present application can be combined with a variety of application scenarios and deployed in various types of applications, so that the application has the function of image style conversion. For example, by combining this solution with an image processing application, the user can re-edit the style of the image through the image processing application; by combining this solution with an image shooting application, the user can freely choose an image style when taking a photo and directly take a photo with the image style. In the above-mentioned multiple application scenarios, the application of the model training method provided by the embodiment of the present application can improve the model performance of the image generation model when performing image style conversion tasks, and when the initial image is converted with a large style span, a good transition can be achieved between the initial image and the newly generated target image, thereby improving the authenticity and image quality of the newly generated target image.

[0065] Figure 2 This is a flowchart of a training method for an image generation model provided in an embodiment of the present application. This method can be applied to the above-mentioned terminal or server, and both the terminal and the server can be regarded as a computer device. In this embodiment of the present application, the training method for the image generation model is introduced with the computer device as the execution subject, see Figure 2 In one possible implementation, the embodiment includes the following steps:

[0066] 201. A computer device obtains an image generation model to be trained, where the image generation model is used to convert any type of style information into corresponding style features, and convert the style of any image based on the style features. The style information is information used to describe the corresponding style.

[0067] In the embodiments of the present application, the style information includes an image or a style tag. Taking an image as an example, the style information is the style presented by the image, for example, the style information is a sketch style, an oil painting style, etc., or the style information is a male face image style, a female face image style, etc.; taking a style tag as an example, the style information is the information indicated by the style tag. It should be noted that the embodiments of the present application do not limit the specific content of the style information.

[0068] In one possible implementation, the image generation model is a model constructed based on GAN (Generative AdversarialNet, generative adversarial network), for example, the image generation model is a StarGAN v2 model. Exemplarily, the image generation model includes a feature encoding network and a generation network. The feature encoding network is used to obtain corresponding style features based on style information. Optionally, the feature encoding network includes two sub-networks, namely a feature extraction network and a feature mapping network. The feature extraction network is used to extract features from the image to obtain style features, and the feature mapping network is used to map style labels to corresponding style features; the generation network is used to perform style conversion on the image based on the acquired style features to generate a new image. It should be noted that the above description of the structure of the image generation model is only an exemplary description. The embodiment of the present application does not limit the structure of the image generation model and the method for the image generation model to perform style conversion on the image.

[0069] 202. The computer device obtains at least two groups of style information, where style information in the same group corresponds to the same style.

[0070] In one possible implementation, taking the example of the style information including images, the computer device obtains at least two groups of images, each group of images corresponding to a style. For example, the computer device obtains a group of male face images and a group of female face images. Taking the example of the style information including style tags, the computer device obtains at least two groups of style tags, each group of style tags corresponding to a style. For example, the computer device obtains a group of tags for describing male facial features and a group of tags for describing female facial features.

[0071] 203. The computer device converts each style information into a corresponding style feature through the image generation model.

[0072] In one possible implementation, the computer device can determine a method for acquiring style features based on the acquired style information. Exemplarily, if the style information includes an image, the computer device extracts features from the image through a feature extraction network in the image generation model, and uses the extracted image features as the style features corresponding to the style information; if the style information includes a style label, the computer device encodes the style label through a feature mapping network in the image generation model to obtain the style features. The style features are represented in the form of vectors or matrices, which is not limited in the embodiments of the present application. It should be noted that the embodiments of the present application do not limit the method for acquiring the style features.

[0073] 204. The computer device obtains a first error and a second error based on the multiple style features obtained by conversion, the distribution information of the multiple style features, and the target distribution information. The target distribution information is used to limit the spatial distribution area of ​​the multiple style features. The first error is used to indicate the error between the distribution information of the multiple style features and the target distribution information. The second error is used to indicate the difference in spatial distribution of style features corresponding to different styles.

[0074] The target distribution information is set by the developer. Exemplarily, the target distribution information is the prior Gaussian distribution N(0, I), that is, a normal distribution with an expectation of 0 and a variance of I. The distribution space indicated by the target distribution information is a compact distribution space, that is, the distribution space indicated by the target distribution information is a distribution space of a smaller area. In an embodiment of the present application, by setting a compact space constraint, that is, obtaining a first error between the distribution information of multiple style features and the target distribution information, and applying the first error to subsequent model training steps, the spatial distribution of style features can be constrained within a compact space, thereby enabling a smooth transition between different styles.

[0075] The second error is used to indicate the spatial distribution of style features corresponding to different styles. In this embodiment of the present application, by setting a separability constraint for style feature clusters, that is, obtaining the second error between style features belonging to different styles and applying this second error to subsequent model training steps, the separability between style feature clusters belonging to different styles can be ensured.

[0076] 205. The computer device adjusts the parameters of the image generation model based on the first error and the second error.

[0077] In one possible implementation, the computer device backpropagates the first and second errors to the image generation model, solves for parameters of each computational layer in the image generation model, and terminates model training until the image generation model meets a model convergence condition. The trained image generation model is then acquired. The model convergence condition is set by the developer and is not limited in this embodiment of the present application.

[0078] The technical solution provided by the embodiment of the present application obtains multiple style features belonging to different styles through an image generation model, determines a first error and a second error based on the spatial distribution of the multiple style features, and applies the first error and the second error to train the image generation model, wherein the application of the first error can constrain the spatial distribution of the style features obtained by the image generation model, so that the distribution of the multiple style features is within a compact space, thereby enabling a smooth transition between the style features belonging to different styles, and the application of the second error can enable the style features belonging to different styles to be separable, that is, through this model training method, the style features belonging to different styles can maintain both distinguishability and smooth transition, so that when the image generation model performs a conversion of the initial image with a large style span, it ensures a good transition between the input image and the newly generated image, thereby improving the authenticity and image quality of the newly generated image.

[0079] The above embodiment is a brief introduction to the embodiment of the present application. Figure 3 This is a schematic diagram of an image generation model provided in an embodiment of the present application, see Figure 3 The image generation model includes a feature extraction network 301, a feature mapping network 302, and a generation network 303. Optionally, the image generation model also includes a discriminant network 304. The discriminant network 304 is used to classify the image output by the generation network 303 and determine whether the image is a real image or an image generated by the model, thereby achieving the effect of detecting the image quality of the image output by the generation network 303. In the embodiment of the present application, the image generation model is trained in an unsupervised training method, that is, the correct image that the image generation model should generate is not provided during the model training process, and there is no real target result as supervision data. Figure 4 This is a flowchart of a training method for an image generation model provided in an embodiment of the present application. Figure 3 As an example, the image generation model shown in Figure 4 The training process of the image generation model is described. In one possible implementation, the embodiment includes the following steps:

[0080] 401. A computer device obtains an image generation model to be trained, at least two sets of style information, and an initial image to be processed, and inputs the at least two sets of style information and the initial image into the image generation model.

[0081] In one possible implementation, the computer device responds to a model training instruction and obtains an image generation model to be trained and training data, wherein the training data is at least two sets of style information and an initial image to be processed. In an embodiment of the present application, the style information is used to indicate the image style, and the style information can be represented in a variety of forms, for example, the style information is represented by an image or by a style label. In one possible implementation, if the style information includes an image, the computer device randomly selects one or more images from the multiple images obtained as the initial image. Of course, the computer device can also obtain the initial image in addition, and the embodiment of the present application is not limited to this. In one possible implementation, if the style information is a style label, the computer device obtains at least two sets of style labels and an initial image to be processed. In some embodiments, the style label includes a style index value d, and different style index values ​​can distinguish different styles. Optionally, the style label also includes random Gaussian noise z, where z satisfies the distribution N(0, I), where the value of I is set by the developer. In the subsequent process of obtaining style features based on style information, the computer device can encode the random Gaussian noise z based on the style index value to obtain style features. In this case, the style features mapped by the random Gaussian noise corresponding to the same style index value belong to the same style, but due to the introduction of random Gaussian noise, these style features may be different. For example, for three sets of data (z1, j), (z2, j), and (z3, j), all have the same style index value j and belong to the same style. However, because z1, z2, and z3 are randomly sampled from the Gaussian noise z, the data z1, z2, and z3 are different. Therefore, the image generation model can obtain three different style features belonging to the same style, and then generate three images belonging to the same style, but these three images may differ in details. For example, if the style index value j represents a "male face", when the image generation model performs style conversion on the initial image based on (z1, j), (z2, j), and (z3, j), it can obtain three different male face images, for example, with different skin colors and hairstyles in the three images.

[0082] It should be noted that the embodiment of the present application does not limit the number of initial images obtained by the computer device. In the embodiment of the present application, only one initial image is used as an example for description.

[0083] In one possible implementation, the computer device pre-processes the style information and the initial image before inputting them into the image generation model. For example, the image is scaled according to the actual situation to adjust the image to a reference size. The reference size is set by the developer, and the embodiment of the present application does not limit this. Alternatively, the computer device performs data enhancement on the style information. For example, if the style information includes an image, the image is rotated, noise is added, etc. The embodiment of the present application does not limit the method of data enhancement. It should be noted that the above description of the method for pre-processing the style information and the initial image is only an exemplary description of a possible implementation method. The embodiment of the present application does not limit the data and processing methods.

[0084] 402. The computer device converts each piece of style information into a corresponding style feature through the image generation model.

[0085] In one possible implementation, in response to the style information including an image, the computer device extracts style features of the image through a feature extraction network in the image generation model. In one possible implementation, the feature extraction network includes multiple convolutional layers for extracting image features. The feature extraction network includes K output branches, where K represents the number of styles corresponding to the image generation model. Each output branch can output a feature representation of the image in a style. Optionally, the feature extraction network utilizes the style encoder in the StarGAN v2 model. In one possible implementation, the computer device sequentially extracts features from the style information through the multiple convolutional layers in the feature extraction network. That is, the computer device sequentially performs convolution operations on the image through the multiple convolutional layers to obtain image features. Each output branch in the feature extraction network outputs a sub-feature, each sub-feature being a feature representation of the image in a particular style. The computer device obtains the sub-features output by each branch to form the style features corresponding to the image. Exemplarily, if each sub-feature is represented as a numerical value, the style feature is represented as a style vector; if each sub-feature is represented as a vector, the style feature is represented as a style matrix. It should be noted that the embodiments of the present application do not limit the structure of the feature extraction network and the method by which the feature extraction network extracts style features.

[0086] In one possible implementation, in response to the style information including a style label, the computer device maps the style label to a corresponding style feature through a feature mapping network in the image generation model. In one possible implementation, the feature mapping network includes an MLP (Multilayer Perceptron) with K output branches. Optionally, the feature mapping network adopts a mapping network in the StarGAN v2 model. In one possible implementation, the computer device encodes the style information through the MLP in the feature mapping network, that is, encodes the random Gaussian noise z based on the style index value d to obtain the style feature. It should be noted that the embodiment of the present application does not limit the method for obtaining style features through the feature mapping network.

[0087] 403. The computer device converts the style of the initial image to be processed based on any style feature through the generation network in the image generation model to obtain a target image.

[0088] Among them, the generative network can generate the target image based on the input initial image x Wherein, s represents any style feature, and the style of the target image is consistent with the style indicated by any style feature. In one possible implementation, the generative network adopts the generator (Generator) in the StarGAN v2 model, and the generative network includes four downsampling layers, four intermediate layers and four upsampling layers, and these operation layers can all be ResBlock (residual network module). In one possible implementation, the computer device applies the AdaIN (Adaptive Instance Normalization) algorithm, inputs the style feature into the generative network, and performs style conversion on the initial image based on the style feature through multiple operation layers in the generative network. Exemplarily, the computer device interpolates the style feature and the image feature of the initial image through the generative network to obtain a new image feature, and then generates the target image based on the new image feature. It should be noted that the structure of the generative network and the method of generating the target image in the embodiment of the present application are not limited.

[0089] 404. The computer device obtains a first error and a second error based on the multiple style features obtained by conversion, the distribution information of the multiple style features, and the target distribution information.

[0090] In one possible implementation, the computer device obtains the KL divergence (Kullback–Leibler divergence) between the distribution information of the multiple style features and the target distribution information, and then determines the first error based on the KL divergence. The target distribution information is used to indicate an expected distribution space. In the embodiment of the present application, the target distribution information is a Gaussian prior distribution N(0, I), that is, it is expected that the multiple style features are distributed in a compact space, so as to achieve a good transition effect between style features belonging to different styles. The KL divergence can be used to measure the difference between the distribution information of the multiple style features and the target distribution information. The larger the value of the KL divergence, the greater the difference between the distribution information of the multiple style features and the target distribution information. The smaller the value of the KL divergence, the smaller the difference between the distribution information of the multiple style features and the target distribution information. In the embodiment of the present application, the first error is positively correlated with the KL divergence. Exemplarily, the method for obtaining the first error is expressed as the following formula (1):

[0091] L kl =E s [D kl (p(s)||N(0,I))] (1)

[0092] Among them, L kl The value of is the first error, D kl (p||q) represents the KL divergence between p and q; s represents the style feature; p(s) represents the distribution information of the style feature s; N(0, I) represents the target distribution information, and I can be set by the developer; E s Express expectations.

[0093] In one possible implementation, the computer device obtains a first distance between any two style features belonging to the same style and a second distance between any two style features belonging to different styles, and then determines the second error based on the difference between the first distance and the second distance corresponding to the same style feature. In one possible implementation, the method for obtaining the second error is expressed as the following formula (2):

[0094] L tril =E (Sa,Sp,Sn) [max(||s a -s p ||-||s a -s n ||+α,0)] (2)

[0095] Among them, L tri The value of is the second error; s a 、s p For style features belonging to the same style, sa 、s n are the stylistic features belonging to different styles; ||s a -s p || represents the first distance, ||s a -s n || represents the second distance; α is a constant, and its value is set by the developer; E (Sa,Sp,Sn) Express expectations.

[0096] It should be noted that the above description of the method for obtaining the first error and the second error is merely an exemplary description of a possible implementation method, and the present embodiment does not limit the method used to obtain the first error and the second error. In the present embodiment, by obtaining the first error and the second error, that is, by setting two constraints, namely, the compact space constraint and the style feature cluster separability constraint, on the one hand, multiple style features are compressed into a compact distribution space, allowing a smooth transition between style features belonging to different styles, and on the other hand, the distinguishability between style features belonging to different styles is maintained.

[0097] 405. The computer device determines a third error based on the similarity between the target image and the initial image.

[0098] In one possible implementation, the computer device uses the LPIPS (Learned Perceptual Image Patch Similarity) method to determine the similarity between the target image and the initial image, and then determines the third error. Exemplarily, first, the computer device performs feature extraction of different scales on the target image and the initial image through at least two scale feature extraction layers, and obtains at least two first features corresponding to the target image and at least two second features corresponding to the initial image. That is, for the target image, the computer device can obtain first features of multiple scales, and for the initial image, the computer device can obtain second features of multiple scales. Then, the computer device obtains the third distance between the first feature and the second feature of the same scale, respectively. Optionally, the third distance is the L2 distance between the first feature and the second feature. Finally, the computer device determines the third error based on the at least two third distances obtained. For example, the computer device takes the average of the at least two third distances obtained to obtain the third error. In one possible implementation, the method for obtaining the third error is expressed as the following formula (3):

[0099] L cont =E s,x [ψ(x,G(x,s))] (3)

[0100] Among them, Lcont The value of is the third error; x represents the initial image, G(x, s) represents the target image, and s represents the style feature; Ψ(x, G(x, s)) represents the degree of visual similarity between the target image and the initial image. In the embodiment of the present application, the LPIPS method is used to obtain Ψ(x, G(x, s)); E s,x Express expectations.

[0101] It should be noted that the above description of the method for obtaining the third error is only an exemplary description of a possible implementation method, and the embodiment of the present application does not limit which method is used to obtain the third error. In the embodiment of the present application, by setting the image content similarity maintenance constraint condition, that is, by obtaining the third error, the third error indicates the similarity in visual perception between the initial image and the target image generated by the model. When the third error is applied to the subsequent model training step, the trained image generation model can maintain the identity feature invariance of the input image during the image generation process, that is, the initial image and the target image are kept consistent in image content.

[0102] It should be noted that the above step 405 of obtaining the third error is an optional step. In some embodiments, only the first error and the second error may be used to train the image generation model.

[0103] 406. The computer device adjusts the parameters of the image generation model based on the first error, the second error, and the third error.

[0104] In one possible implementation, the computer device performs weighted processing on the first error, the second error, and the third error to obtain a fourth error, and adjusts the parameters of the image generation model based on the fourth error. Exemplarily, the fourth error is obtained as shown in the following formula (4):

[0105] L smooth =λ kl L kl +λ tri L tri +λ cont L cont (4)

[0106] Among them, L smooth The value of is the fourth error, L kl The value of is the first error, L tri The value of is the second error, L cont The value of is the third error; kl ,λ tri and λ contIt is a hyperparameter used to adjust the weight of each error, and its value can be set by the developer. In one possible implementation, the computer device can apply the backpropagation method to update the parameters of the image generation model. For example, the computer device solves the various parameters in the image generation model based on the gradient descent method of the Adam (Adaptove moment estimation) algorithm. It should be noted that the embodiment of the present application does not limit the specific method for updating the parameters of the image generation model. In one possible implementation, after the computer device completes the update of the parameters of the image generation model, if the image generation model meets the model convergence condition, the trained image generation model is obtained; if the model convergence adjustment is not met, the next batch of training data is read, and the above steps 401 to 406 are continued. Among them, the model convergence condition is set by the developer, and the embodiment of the present application does not limit this.

[0107] It should be noted that, in some embodiments, the computer device may not perform weighted operations on the first error, the second error, and the third error, and directly backpropagate the first error, the second error, and the third error to the image generation model to perform model training. This embodiment of the present application is not limited to this.

[0108] In some embodiments, in addition to obtaining the loss functions of the four errors mentioned above, developers can also set other forms of loss functions L exist , obtain other errors, combine the other errors with the fourth error to obtain the fifth error, and perform model training based on the fifth error, that is, backpropagate the fifth error to the image generation model and re-solve the parameters of each operation layer in the image generation model. It should be noted that the embodiment of the present application has L exist The specific loss function is not limited. In one possible implementation, the fifth error is expressed as the following formula (5):

[0109] L new =L smooth +L exist (5)

[0110] Among them, L new The value is the fifth error, L smooth The value of is the fourth error, L exist The values ​​are other errors.

[0111] The technical solution provided by the embodiment of the present application obtains multiple style features belonging to different styles through an image generation model, determines a first error and a second error based on the spatial distribution of the multiple style features, and applies the first error and the second error to train the image generation model, wherein the application of the first error can constrain the spatial distribution of the style features obtained by the image generation model, so that the distribution of the multiple style features is within a compact space, thereby enabling a smooth transition between the style features belonging to different styles, and the application of the second error can enable the style features belonging to different styles to be separable, that is, through this model training method, the style features belonging to different styles can maintain both distinguishability and smooth transition, so that when the image generation model performs a conversion of the initial image with a large style span, it ensures a good transition between the input image and the newly generated image, thereby improving the authenticity and image quality of the newly generated image.

[0112] Figure 5 This is a schematic diagram of the distribution of style features provided in the embodiment of the application. Generally, the distribution of style features is as follows: Figure 5 As shown in Figure (a), the multiple style features 501 and 502 corresponding to different styles lack a good transition in spatial distribution. When the image generation model converts the initial image 503 into the style indicated by the reference image 504, the generated target image 505 has low realism and poor image quality. By applying the image generation model training method provided in the embodiment of the present application, the model is trained by setting two constraints: compact space constraint and style feature cluster separability constraint. This can make the style features 506 and 507 corresponding to different styles have a smooth transition in spatial distribution and maintain the distinguishability, as shown in Figure 5a. Figure 5 As shown in FIG. 5( b ), the target image 508 generated by the image generation model has high realism and high image quality. Figure 6 This is a performance comparison diagram of an image generation model obtained by applying different model training methods provided in an embodiment of the present application. When the image generation model obtained by applying training methods (a), (b), (c), and (d) performs the image style conversion task, the target images generated by the image generation model are respectively as follows: Figure 6 As shown in the first, second, third and fourth rows of FIG, the training method (d) is the model training method provided in the embodiment of the present application, such as Figure 6 The rendering shown in the fourth row shows that the image generation model trained by the model training method provided in the embodiment of the present application has good model performance when performing style conversion on images. The authenticity and image quality of each target image generated are high, and the content consistency between each image can be maintained.

[0113] The above embodiment introduces a method for training an image generation model. The image generation obtained by applying the above model training method can be deployed in various types of applications and combined with various application scenarios. In the embodiments of this application, the application of the image generation model in an image processing application is used as an example for illustration. In one possible implementation, the process of applying the image generation model to convert the image style and generate a new image includes the following steps.

[0114] Step 1: In response to an image generation operation, the terminal obtains a first image to be processed and style conversion requirement information, and sends the first image and the style conversion requirement information to a server.

[0115] The terminal is a device used by a user, which has installed and runs a target application that can provide image style conversion functionality. The server is the background server of the target application, which is equipped with a trained image generation model trained using the above-mentioned model training method. The style conversion requirement is used to indicate the goal of performing style conversion on the first image. In this embodiment of the present application, the style conversion requirement information can be expressed in various ways. For example, the style conversion requirement information is expressed by an image or by a style tag, which is not limited in this embodiment of the present application.

[0116] In one possible implementation, the terminal displays an image editing interface of the target application, which includes a first upload entry for an image to be processed, through which a user can upload the first image to be processed. Optionally, the image editing interface includes a second upload entry for a reference image, which is used to indicate the image style conversion requirement. That is, the style presented by the reference image is the style conversion target of the first image. Optionally, the image editing interface includes multiple style selection controls, each corresponding to a style. The style selection controls can be presented in the form of an image or a text label, which is not limited in this embodiment of the present application. When a user selects a style selection control, the image style indicated by the style selection control is selected, which means that a style conversion requirement is provided. Optionally, the image editing interface includes an information input area, in which the user can enter the style conversion requirement. It should be noted that the embodiment of the present application does not limit the method for obtaining the style conversion requirement.

[0117] Step 2: The server calls the trained image generation model and performs style transformation on the first image based on the style transformation requirement information to generate a second image.

[0118] In the embodiment of the present application, the image style transfer process is described by taking the StarGAN v2 model as an example. In one possible implementation, in response to receiving the first image and style transfer requirement information, the server may determine a style feature extraction method based on the representation of the style transfer requirement information. For example, if the style transfer requirement information is an image, the server extracts features from the image through the feature extraction network in the image generation model to obtain style features. If the style transfer information is a style label, the server encodes the style label into a style feature through the feature mapping network in the image generation model. After obtaining the style features, the server inputs the style features and the first image into the generation network in the image generation model. The generation network transforms the style of the first image into the style indicated by the style features. For example, the server interpolates the style features and the image features of the first image through the generation network to obtain a new image feature. The server then generates a second image based on the new image feature, that is, outputs the image after the style transfer of the first image. It should be noted that the above description of the method for generating the second image is only an exemplary description of a possible implementation method, and the embodiment of the present application does not limit the specific method used to generate the second image.

[0119] Step 3: The server sends the second image to the terminal, and the terminal displays the second image.

[0120] In a possible implementation, the terminal displays an image display interface of the target application. After receiving the second image, the terminal displays the second image on the image display interface.

[0121] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0122] Figure 7 This is a structural diagram of a training device for an image generation model provided in an embodiment of the present application, see Figure 7 , the device comprises:

[0123] A model acquisition module 701 is used to acquire an image generation model to be trained. The image generation model is used to convert any type of style information into corresponding style features, and convert the style of any image based on the style features. The style information is information used to describe the corresponding style.

[0124] An information acquisition module 702 is configured to acquire at least two sets of style information, where the style information in the same set corresponds to the same style;

[0125] An information conversion module 703 is configured to convert each style information into a corresponding style feature through the image generation model;

[0126] An error acquisition module 704 is configured to acquire a first error and a second error based on the converted plurality of style features, distribution information of the plurality of style features, and target distribution information, wherein the target distribution information is used to define a spatial distribution region of the plurality of style features, the first error is used to indicate an error between the distribution information of the plurality of style features and the target distribution information, and the second error is used to indicate a difference in spatial distribution of style features corresponding to different styles;

[0127] The parameter adjustment module 705 is configured to adjust the parameters of the image generation model based on the first error and the second error.

[0128] In one possible implementation, the information conversion module 703 is configured to perform any of the following:

[0129] In response to the style information including an image, extracting style features of the image through a feature extraction network in the image generation model;

[0130] In response to the style information including a style label, the style label is mapped to a corresponding style feature through a feature mapping network in the image generation model.

[0131] In one possible implementation, the error acquisition module 704 is used to

[0132] Obtaining the KL divergence between the distribution information of the plurality of style features and the target distribution information;

[0133] determining the first error based on the KL divergence, where the first error is positively correlated with the KL divergence;

[0134] Obtaining a first distance between any two style features belonging to the same style and a second distance between any two style features belonging to different styles;

[0135] The second error is determined based on a difference between the first distance and the second distance corresponding to the same style feature.

[0136] In one possible implementation, the device further includes:

[0137] The image acquisition module is used to convert the style of the initial image to be processed based on any style feature through the generation network in the image generation model to obtain a target image, where the style of the target image is consistent with the style indicated by any style feature.

[0138] In one possible implementation, the device further includes:

[0139] an error determination module, configured to determine a third error based on a similarity between the target image and the initial image;

[0140] The parameter adjustment module 705 is used to adjust the parameters of the image generation model based on the first error, the second error and the third error.

[0141] In one possible implementation, the error determination module is configured to:

[0142] Performing feature extraction of different scales on the target image and the initial image through at least two scale feature extraction layers, respectively, to obtain at least two first features corresponding to the target image and at least two second features corresponding to the initial image;

[0143] respectively obtaining a third distance between the first feature and the second feature of the same scale;

[0144] The third error is determined based on the acquired at least two third distances.

[0145] In one possible implementation, the device further includes:

[0146] a weighted processing module, configured to perform weighted processing on the first error, the second error, and the third error to obtain a fourth error;

[0147] The parameter adjustment module 705 is configured to adjust parameters of the image generation model based on the fourth error.

[0148] The device provided in the embodiment of the present application obtains multiple style features belonging to different styles through an image generation model, determines a first error and a second error based on the spatial distribution of the multiple style features, and applies the first error and the second error to train the image generation model, wherein the application of the first error can constrain the spatial distribution of the style features obtained by the image generation model, so that the distribution of the multiple style features is within a compact space, thereby enabling a smooth transition between the style features belonging to different styles, and the application of the second error can enable the style features belonging to different styles to be separable, that is, the application of the above-mentioned model training device can enable the style features belonging to different styles to maintain both distinguishability and smooth transition, so that when the image generation model performs a conversion of the initial image with a large style span, it ensures a good transition between the input image and the newly generated image, thereby improving the authenticity and image quality of the newly generated image.

[0149] It should be noted that the image generation model training device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate the training of the image generation model. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image generation model training device provided in the above embodiment and the image generation model training method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0150] The computer device provided by the above technical solution can be implemented as a terminal or a server, for example, Figure 8 The figure is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. Terminal 800 may be a smartphone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, or a desktop computer. Terminal 800 may also be referred to as user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other similar names.

[0151] Typically, the terminal 800 includes one or more processors 801 and one or more memories 802 .

[0152] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0153] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one computer program, which is used to be executed by the processor 801 to implement the training method of the image generation model provided in the method embodiment of the present application.

[0154] In some embodiments, terminal 800 may optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 803 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, a positioning assembly 808, and a power supply 809.

[0155] The peripheral device interface 803 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802, and the peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802, and the peripheral device interface 803 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0156] The RF circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 804 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 804 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 804 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0157] Display screen 805 is used to display a user interface (UI). This UI can include graphics, text, icons, videos, or any combination thereof. When display screen 805 is a touchscreen display, it can also capture touch signals on or above the surface of display screen 805. These touch signals can be input as control signals to processor 801 for processing. Display screen 805 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be a single display screen 805, located on the front panel of terminal 800. In other embodiments, there can be at least two display screens 805, located on different surfaces of terminal 800 or in a foldable design. In some embodiments, display screen 805 can be a flexible display, located on a curved or foldable surface of terminal 800. Display screen 805 can also be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 805 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0158] The camera assembly 806 is used to capture images or videos. Optionally, the camera assembly 806 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0159] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input into the processor 801 for processing, or input into the radio frequency circuit 804 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, each located in different parts of the terminal 800. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0160] Positioning component 808 is used to locate the current geographic location of terminal 800 to implement navigation or LBS (Location Based Service). Positioning component 808 can be based on the US GPS (Global Positioning System), China's Beidou system, Russia's Greninja system, or the European Union's Galileo system.

[0161] Power supply 809 is used to power various components in terminal 800. Power supply 809 can be AC ​​power, DC power, disposable batteries, or rechargeable batteries. When power supply 809 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0162] In some embodiments, the terminal 800 further includes one or more sensors 810 , including but not limited to: an acceleration sensor 811 , a gyroscope sensor 812 , a pressure sensor 813 , a fingerprint sensor 814 , an optical sensor 815 , and a proximity sensor 816 .

[0163] The accelerometer 811 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 800. For example, the accelerometer 811 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 801 can control the display screen 805 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 811. The accelerometer 811 can also be used to collect game or user motion data.

[0164] The gyroscope sensor 812 can detect the orientation and rotation angle of the terminal 800. It can work in conjunction with the accelerometer 811 to collect the user's 3D movements of the terminal 800. Based on the data collected by the gyroscope sensor 812, the processor 801 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0165] The pressure sensor 813 can be set on the side frame of the terminal 800 and / or the lower layer of the display screen 805. When the pressure sensor 813 is set on the side frame of the terminal 800, it can detect the user's grip signal of the terminal 800, and the processor 801 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 813. When the pressure sensor 813 is set on the lower layer of the display screen 805, the processor 801 controls the operational controls on the UI interface based on the user's pressure operation on the display screen 805. The operational controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0166] The fingerprint sensor 814 is used to collect the user's fingerprint. The processor 801 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 814, or the fingerprint sensor 814 identifies the user's identity based on the collected fingerprint. When the user's identity is recognized as a trusted identity, the processor 801 authorizes the user to perform relevant sensitive operations, such as unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 814 can be set on the front, back, or side of the terminal 800. When a physical button or manufacturer logo is provided on the terminal 800, the fingerprint sensor 814 can be integrated with the physical button or manufacturer logo.

[0167] The optical sensor 815 is used to detect ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the display screen 805 based on the ambient light intensity detected by the optical sensor 815. Specifically, when the ambient light intensity is high, the display brightness of the display screen 805 is increased; when the ambient light intensity is low, the display brightness of the display screen 805 is decreased. In another embodiment, the processor 801 can also dynamically adjust the shooting parameters of the camera assembly 806 based on the ambient light intensity detected by the optical sensor 815.

[0168] Proximity sensor 816, also known as a distance sensor, is typically located on the front panel of terminal 800. Proximity sensor 816 is used to detect the distance between the user and the front of terminal 800. In one embodiment, when proximity sensor 816 detects that the distance between the user and the front of terminal 800 is gradually decreasing, processor 801 controls display screen 805 to switch from the screen-on state to the screen-off state. When proximity sensor 816 detects that the distance between the user and the front of terminal 800 is gradually increasing, processor 801 controls display screen 805 to switch from the screen-off state to the screen-on state.

[0169] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the terminal 800, and the terminal 800 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0170] Figure 9 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 901 and one or more memories 902, wherein the one or more memories 902 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 901 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server 900 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 900 may also include other components for implementing device functions, which will not be described in detail here.

[0171] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program. The at least one computer program can be executed by a processor to implement the image generation model training method described in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0172] In an exemplary embodiment, a computer program product is also provided, comprising at least one computer program stored in a computer-readable storage medium. A processor of a computer device reads the at least one computer program from the computer-readable storage medium and executes the at least one computer program, causing the computer device to implement the operations performed by the method for training an image generation model.

[0173] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0174] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A training method for an image generation model, characterized in that: The method comprises: Obtaining an image generation model to be trained, wherein the image generation model is used to convert any type of style information into corresponding style features, and convert the style of any image based on the style features, wherein the style information is information used to describe the corresponding style; Obtain at least two sets of style information, where the style information in the same set corresponds to the same style; Convert each style information into a corresponding style feature through the image generation model; Obtaining a first error and a second error based on the converted multiple style features, distribution information of the multiple style features, and target distribution information, wherein the target distribution information is used to define a spatial distribution area of ​​the multiple style features, the first error is used to indicate an error between the distribution information of the multiple style features and the target distribution information, and the second error is used to indicate a difference in spatial distribution of style features corresponding to different styles; Based on the first error and the second error, parameters of the image generation model are adjusted.

2. The method according to claim 1, characterized in that The converting each style information into a corresponding style feature by the image generation model includes any of the following: In response to the style information including an image, extracting style features of the image through a feature extraction network in the image generation model; In response to the style information including a style label, the style label is mapped to a corresponding style feature through a feature mapping network in the image generation model.

3. The method according to claim 1, characterized in that The obtaining of a first error and a second error based on the plurality of style features obtained by conversion, the distribution information of the plurality of style features, and the target distribution information includes: Obtaining a KL divergence between the distribution information of the plurality of style features and the target distribution information; determining the first error based on the KL divergence, where the first error is positively correlated with the KL divergence; Obtaining a first distance between any two style features belonging to the same style and a second distance between any two style features belonging to different styles; The second error is determined based on a difference between the first distance and the second distance corresponding to the same style feature.

4. The method according to claim 1, wherein After converting each piece of style information into a corresponding style feature through the image generation model, the method further includes: The style of the initial image to be processed is converted based on any style feature through the generation network in the image generation model to obtain a target image, and the style of the target image is consistent with the style indicated by any style feature.

5. The method according to claim 4, characterized in that After converting the style of the initial image to be processed based on any style feature through the generative network in the image generation model to obtain the target image, the method further includes: determining a third error based on a similarity between the target image and the initial image; The adjusting the parameters of the image generation model based on the first error and the second error includes: Based on the first error, the second error, and the third error, parameters of the image generation model are adjusted.

6. The method according to claim 5, characterized in that The determining of a third error based on the similarity between the target image and the initial image includes: Performing feature extraction of different scales on the target image and the initial image through at least two scale feature extraction layers, respectively, to obtain at least two first features corresponding to the target image and at least two second features corresponding to the initial image; respectively obtaining a third distance between the first feature and the second feature of the same scale; The third error is determined based on the acquired at least two third distances.

7. The method according to claim 5, characterized in that After determining the third error based on the similarity between the target image and the initial image, the method further includes: performing weighted processing on the first error, the second error, and the third error to obtain a fourth error; The adjusting the parameters of the image generation model based on the first error, the second error, and the third error includes: Based on the fourth error, parameters of the image generation model are adjusted.

8. A training device for an image generation model, characterized in that: The device comprises: A model acquisition module, configured to acquire an image generation model to be trained, wherein the image generation model is configured to convert any type of style information into corresponding style features, and to convert the style of any image based on the style features, wherein the style information is information used to describe the corresponding style; An information acquisition module, configured to acquire at least two sets of style information, wherein the style information in the same set corresponds to the same style; An information conversion module, configured to convert each style information into a corresponding style feature through the image generation model; an error acquisition module, configured to acquire a first error and a second error based on a plurality of style features obtained through conversion, distribution information of the plurality of style features, and target distribution information, wherein the target distribution information is used to define a spatial distribution region of the plurality of style features, the first error is used to indicate an error between the distribution information of the plurality of style features and the target distribution information, and the second error is used to indicate a difference in spatial distribution of style features corresponding to different styles; A parameter adjustment module is used to adjust the parameters of the image generation model based on the first error and the second error.

9. A computer device, characterized in that: The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the operations performed by the training method of the image generation model according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the operations performed by the image generation model training method according to any one of claims 1 to 7.

Images