An image style conversion method, device, computer equipment and storage medium

Abstract

Embodiments of the present application disclose an image style conversion method and device, computer equipment and a storage medium. Embodiments of the present application can obtain a basic image style conversion model, a target image style conversion model, a basic image style feature, a target image style feature and an image to be converted. The model parameters of the basic image style conversion model are used to adjust the model parameters of the target image style conversion model, and an adjusted target image style conversion model is obtained. The basic image style feature and the target image style feature are subjected to feature fusion processing, and a fused image style feature is obtained. The adjusted target image style conversion model and the fused image style feature are used to perform style conversion processing on the image to be converted, and a target image conforming to the target image style is obtained, thereby improving the accuracy of image style conversion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an image style conversion method and device, computer equipment and storage medium. BACKGROUND

[0002] With the rapid development of communication and computer technology, the image processing technology based on computer and communication has also developed stably and rapidly, and is applied to various fields. For example, the image processing technology can be used to convert the style of an image to obtain an image with different style, etc. In the research and practice of the prior art, the present inventors have found that the prior art has the problem of inaccurate image style conversion when using the image processing technology to convert the style of an image. SUMMARY

[0003] The embodiments of the present application provide an image style conversion method, device and computer equipment, which improves the accuracy of image style conversion.

[0004] The embodiments of the present application provide an image style conversion method, which comprises the following steps:

[0005] Obtaining a target style image to be processed, performing quality adjustment processing on the target style image to be processed, and obtaining an adjusted target style image;

[0006] Performing feature extraction on the adjusted target style image to obtain a target image style feature;

[0007] Obtaining a basic image style conversion model, performing migration training on a preset target image style conversion model by using the basic image style conversion model and the target image style feature, and obtaining a target image style conversion model;

[0008] Adjusting the model parameters of the target image style conversion model based on the model parameters of the basic image style conversion model to obtain an adjusted target image style conversion model;

[0009] Obtaining a basic image style feature and an image to be converted, and performing feature fusion processing on the basic image style feature and the target image style feature to obtain a fused image style feature;

[0010] Performing style conversion processing on the image to be converted by using the adjusted target image style conversion model and the fused image style feature to obtain a target image conforming to a target image style.

[0011] Correspondingly, the embodiments of the present application also provide an image style conversion device, which comprises:

[0012] The acquisition unit is configured to acquire a target style image to be processed, and perform quality optimization processing on the target style image to be processed to obtain an optimized target style image.

[0013] The feature extraction unit is configured to perform feature extraction on the optimized target style image to obtain a target image style feature.

[0014] The migration training unit is configured to acquire a base image style conversion model and the target image style feature, and perform migration training on a preset target image style conversion model by using the base image style conversion model and the target image style feature to obtain a target image style conversion model.

[0015] The adjustment unit is configured to adjust model parameters of the target image style conversion model based on model parameters of the base image style conversion model to obtain an adjusted target image style conversion model.

[0016] The feature fusion unit is configured to acquire a base image style feature and an image to be converted, and perform feature fusion processing on the base image style feature and the target image style feature to obtain a fused image style feature.

[0017] The style conversion unit is configured to perform style conversion processing on the image to be converted by using the adjusted target image style conversion model and the fused image style feature to obtain a target image conforming to a target image style.

[0018] In an embodiment, the style conversion unit can include:

[0019] The encoding subunit is configured to perform encoding processing on the image to be converted by using the adjusted target image style conversion model to obtain a to-be-converted style feature of the image to be converted.

[0020] The feature conversion subunit is configured to perform feature conversion processing on the to-be-converted style feature by using the fused image style feature to obtain a converted style feature.

[0021] The decoding subunit is configured to perform decoding processing on the converted style feature by using the adjusted target image style conversion model to obtain the target image conforming to the target image style.

[0022] In an embodiment, the encoding subunit can include:

[0023] The feature extraction module is configured to perform feature extraction processing on the image to be converted by using the adjusted target image style conversion model to obtain feature information of the image to be converted.

[0024] The distribution mapping module is configured to perform distribution mapping processing on the feature information by using the adjusted target image style conversion model to obtain distribution features of the feature information.

[0025] The generation module is configured to generate a to-be-converted style feature of the to-be-converted image according to the distribution features by using the adjusted target image style conversion model.

[0026] In an embodiment, the feature fusion unit can include:

[0027] The statistical subunit is configured to perform statistical processing on the basic image style features to obtain statistical image style features.

[0028] The fusion subunit is configured to perform fusion processing on the statistical image style features and the target image style features to obtain fused image style features.

[0029] In an embodiment, the style conversion unit can include:

[0030] The first screening subunit is configured to screen at least one to-be-adjusted functional layer from the target image style conversion model.

[0031] The second screening subunit is configured to screen a corresponding adjustment reference layer from the basic image style conversion model according to the to-be-adjusted functional layer.

[0032] The parameter adjustment subunit is configured to adjust parameters of the to-be-adjusted functional layer based on parameters of the adjustment reference layer to obtain the adjusted target image style conversion model.

[0033] In an embodiment, the parameter adjustment subunit can include:

[0034] The determination module is configured to determine a parameter fusion strength of the to-be-adjusted functional layer and a parameter fusion strength of the adjustment reference layer.

[0035] The parameter fusion module is configured to perform parameter fusion processing on the parameters of the to-be-adjusted functional layer and the parameters of the adjustment reference layer according to the parameter fusion strength of the to-be-adjusted functional layer and the parameter fusion strength of the adjustment reference layer to obtain the adjusted target image style conversion model.

[0036] In an embodiment, the migration training unit can include:

[0037] The initialization subunit is configured to initialize model parameters of the preset target image style conversion model by using model parameters of the basic image style conversion model to obtain an initialized target image style conversion model.

[0038] The training subunit is configured to train the initial target image style conversion model by using the target image style feature, to obtain the target image style conversion model.

[0039] In an embodiment, the training subunit can include:

[0040] The acquisition model is configured to acquire a training image.

[0041] The style conversion module is configured to perform style conversion processing on the training image by using the target image style feature and the initial target image style conversion model, to obtain a style-converted image.

[0042] The calculation module is configured to calculate loss information of the style-converted image and a preset target style image.

[0043] The adjustment module is configured to adjust model parameters of the initial target image style conversion model according to the loss information, to obtain the basic image style conversion model.

[0044] In an embodiment, the style conversion module can include:

[0045] The encoding sub-module is configured to encode the training image by using the initial target image style conversion model, to obtain a training image style feature.

[0046] The feature conversion sub-module is configured to perform feature conversion processing on the training image style feature by using the target image style feature, to obtain a style-converted feature of the training image.

[0047] The generation sub-module is configured to generate a style-converted image of the training image according to the style-converted feature by using the initial target image style conversion model.

[0048] In an embodiment, the generation sub-module can be configured to:

[0049] Decode the style-converted feature by using the initial target image style conversion model, to obtain a decoded image.

[0050] Perform semantic segmentation processing on the training image, to obtain semantic information of the training image.

[0051] Perform feature enhancement processing on the decoded image by using the semantic information, to obtain the style-converted image.

[0052] The embodiment of the present application further provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in any of the various optional manners of the above aspect.

[0053] Correspondingly, the embodiment of the present application further provides a storage medium, which stores instructions, and the instructions are executed by a processor to implement the image style conversion method provided in any of the embodiments of the present application.

[0054] The embodiment of the present application can obtain a target style image to be processed, perform quality optimization processing on the target style image to be processed to obtain an optimized target style image, perform feature extraction on the optimized target style image to obtain a target image style feature, obtain a basic image style conversion model and the target image style feature, wherein the basic image style conversion model is used to convert an image style into a basic image style, use the basic image style conversion model and the target image style feature to perform migration training on a preset target image style conversion model to obtain a target image style conversion model, wherein the target image style conversion model is used to convert an image style into a target image style, adjust and process model parameters of the basic image style conversion model and model parameters of the target image style conversion model to obtain an adjusted target image style conversion model, obtain a basic image style feature and an image to be converted, perform feature fusion processing on the basic image style feature and the target image style feature to obtain a fused image style feature, and use the adjusted target image style conversion model and the fused image style feature to perform style conversion processing on the image to be converted to obtain a target image conforming to the target image style, thereby improving the accuracy of image style conversion. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0056] Figure 1 is a scene schematic diagram of the image style conversion method provided by the embodiment of the present application;

[0057] Figure 2 is a flow schematic diagram of the image style conversion method provided by the embodiment of the present application;

[0058] Figure 3is a scene schematic diagram of an image style provided by an embodiment of the present application;

[0059] Figure 4 is a scene schematic diagram of migration training provided by an embodiment of the present application;

[0060] Figure 5 is a scene schematic diagram of training an image adjustment model provided by an embodiment of the present application;

[0061] Figure 6 is an effect schematic diagram of an image adjustment model provided by an embodiment of the present application;

[0062] Figure 7 is a scene schematic diagram of image style conversion provided by an embodiment of the present application;

[0063] Figure 8 is another flow schematic diagram of an image style conversion method provided by an embodiment of the present application;

[0064] Figure 9 is another scene schematic diagram of an image style conversion method provided by an embodiment of the present application;

[0065] Figure 10 is another scene schematic diagram of an image style conversion method provided by an embodiment of the present application;

[0066] Figure 11 is a structure schematic diagram of an image style conversion apparatus provided by an embodiment of the present application;

[0067] Figure 12 is a structure schematic diagram of a terminal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. However, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0069] An image style conversion method is provided by an embodiment of the present application, which can be executed by an image style conversion apparatus. The image style conversion apparatus can be integrated in a computer device. The computer device can include a terminal and a server, etc.

[0070] The terminal can be a smartphone, a tablet computer, a notebook computer, a personal computer (PC), a smart home, a wearable electronic device, a VR / AR device, a vehicle-mounted computer, and the like. The server can be an interworking server between multiple heterogeneous systems or a background server, and can also be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system. The server can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and basic cloud computing services such as big data and artificial intelligence platforms, and the like.

[0071] In an embodiment, as Figure 1 The image style conversion device can be integrated in a computer device such as a terminal or a server to implement the image style conversion method proposed in the embodiments. Specifically, the computer device can obtain a basic image style conversion model and a target image style feature, wherein the basic image style conversion model is used to convert the image style into a basic image style; the preset target image style conversion model is trained by using the basic image style conversion model and the target image style feature, to obtain a target image style conversion model, wherein the target image style conversion model is used to convert the image style into a target image style; the model parameters of the basic image style conversion model and the model parameters of the target image style conversion model are adjusted and processed to obtain an adjusted target image style conversion model; the basic image style feature and the image to be converted are obtained, and the basic image style feature and the target image style feature are fused to obtain a fused image style feature; the image to be converted is processed by using the adjusted target image style conversion model and the fused image style feature, to obtain a target image conforming to the target image style.

[0072] The following will be described in detail, and it should be noted that the order of the following embodiments is not limited to the preferred order of the embodiments.

[0073] The embodiments of the present application will be described from the perspective of an image style conversion device, which can be integrated in a computer device, which can be a server or a terminal or the like.

[0074] As Figure 2 An image style conversion method is provided, and the specific process includes:

[0075] 101, obtaining a target style image to be processed, and performing quality adjustment processing on the target style image to be processed to obtain an adjusted target style image.

[0076] The target style image to be processed includes a target style image that needs to improve the image quality. For example, the target style image to be processed can be a low-resolution image. For another example, the target style image to be processed can be a noisy image, and the like.

[0077] In an embodiment, the target style image to be processed can be processed in quality adjustment in various ways. For example, the preset image adjustment model can process the target style image to be processed in quality adjustment.

[0078] The preset image adjustment model can be a model stacked with Conv-BN-Relu as a network structure. The preset image adjustment model can convert a low-quality image into a high-quality image. For example, the preset image adjustment model can convert a low-resolution image into a high-resolution image, and the like.

[0079] 102. Extract features from the adjusted target style image to obtain target image style features.

[0080] In an embodiment, after obtaining the adjusted target style image, features can be extracted from the adjusted target style image to obtain target image style features.

[0081] The target image style features include information that best reflects the characteristics of the image style.

[0082] In an embodiment, the adjusted target style image can be extracted in various ways to obtain target image style features.

[0083] For example, various machine learning network-based models can be used to extract features from the optimized target style image to obtain target image style features.

[0084] For another example, color channel values on each color channel of the adjusted target style image can be extracted, and then the color channel values on each color channel can be processed respectively to obtain target image style features. For example, convolution operation can be performed on the color channel values on each color channel, and then weighted summation can be performed to obtain target image style features.

[0085] The color channel can be a storage space that stores color components of an image. For example, a red (R) color channel can be a channel that stores red color information; a green (G) color channel can be a channel that stores green color channel values; and a blue (B) color channel can be a channel that stores blue color channel values. Each type of color channel can be an 8-bit storage space, and each bit stores a corresponding color channel value.

[0086] For example, when the adjusted target style image is a red, green, blue (RGB) color mode, then the constituent adjusted target style image can include three color channels of red, green and blue. Then, the color channel values corresponding to the three color channels can be respectively subjected to convolution operation processing, so as to obtain color feature information corresponding to each color channel. Next, the color feature information corresponding to each color channel can be weighted and summed, so as to obtain a target image style feature of the target style image, and so on.

[0087] 103、obtain a basic image style conversion model and a target image style feature, and use the basic image style conversion model and the target image style feature to perform transfer training on a preset target image style conversion model to obtain a target image style conversion model.

[0088] The basic image style conversion model includes a model that can convert the image style of an image into a basic image style.

[0089] In an embodiment, the basic image style conversion model can be a model based on a machine learning network framework.

[0090] Machine learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how a computer simulates or implements human learning behavior to obtain new knowledge or skills, reorganizes existing knowledge structure to continuously improve its own performance. Machine learning is the core of artificial intelligence and the fundamental approach to enabling computers to have intelligence, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning techniques.

[0091] For example, the machine learning network can include Convolutional Neural Networks (CNN), De-Convolutional Networks (DN), Deep Neural Networks (DNN), Deep Convolutional Inverse Graphics Networks (DCIGN), Generative Adversarial Networks (GAN), Region-based Convolutional Networks (RCNN), Faster Region-based Convolutional Networks (Faster RCNN), Bidirectional Encoder Representations from Transformers (BERT) model, and the like.

[0092] For example, the base image style conversion model can be a model based on one of the CNN, DNN, GAN, and the like.

[0093] For example, the image in the embodiments of the present application can include a photo, a painting, a cartoon, a watercolor painting, a map, a film and television screen, and the like.

[0094] For example, the image style can include a representative appearance of the image as a whole.

[0095] In an embodiment, the image style can include various image styles.

[0096] For example, the image created by each person generally has personal characteristics, so the image created by each person generally forms a personal style.

[0097] For example, the painting works of each person generally have personal painting styles. Therefore, the painting works of each person can form their own distinctive style. For example, the painting works of Van Gogh have a distinctive "Van Gogh style"; the painting works of Picasso have a distinctive "Picasso style"; the painting works of Xu Beihong have a distinctive "Xu Beihong style"; the painting works of Hayao Miyazaki have a distinctive "Hayao Miyazaki style", and the like.

[0098] For example, each person's photographic works will have personal characteristics. Therefore, each person's photographic works can form their own distinctive style.

[0099] For example, according to the content of the image, the image can also be divided into various image styles. For example, the image can be divided into abstract style, realistic style, etc.

[0100] For example, according to the material for creating the painting, the painting can also be divided into various styles. For example, the painting can be divided into oil painting style, cartoon style, watercolor style, ink painting style, etc.

[0101] For example, as shown in FIG. 1, image 001, image 002, image 003, image 004, image 005, and image 006 are all images of different styles. Figure 3

[0102] In an embodiment, the basic image style can include the styles of various images. For example, there are 10 image styles, and each of the 10 image styles is different. However, in order to facilitate the representation of the 10 image styles, the 10 image styles can be collectively referred to as a basic image style.

[0103] In an embodiment, the target image style can include the style that the user wants the image to present. For example, the original style of the image to be converted is a realistic style, but the user wants to convert the image to be converted to an abstract style, and the abstract style can be the target image style.

[0104] In an embodiment, the basic image style feature includes information that can best reflect the characteristics of the image style.

[0105] Since various different image styles can be collectively referred to as a basic image style, the basic image style feature can also include the features of various different image styles. For example, the basic image style includes 10 different image styles, and the basic image style feature can include 10 different image style features. For example, the basic image style includes 10 different animation styles, and each animation style has the style of an animation artist, and the basic image style feature can also include 10 different animation style features.

[0106] In an embodiment, before obtaining the basic image style feature, various different styles of images can be encoded using a style encoder to obtain features of various different image styles.

[0107] ​Specifically, the style encoder can be used to encode multiple images respectively, so as to obtain multiple image style features. For example, the style encoder can be used to encode 5000 images. Among the 5000 images, there are 20 styles. For example, after the 5000 images are encoded by the style encoder, 5000 encoding vectors can be obtained. Then, the encoding vectors with the same style can be averaged and summed, so as to obtain 20 basic image style features.

[0108] The style encoder can also be a machine learning network. For example, the style encoder can be a CNN network, a DNN network or a GAN network, etc.

[0109] In an embodiment, the target image style feature includes information that best reflects the characteristics of the image style.

[0110] In an embodiment, the target image style feature and the basic image style feature can have multiple forms of expression. For example, the target image style feature and the basic image style feature can be in the form of vectors or matrices, etc.

[0111] The target image style conversion model includes a model that can convert the image style of an image to a target image style.

[0112] In an embodiment, the target image style conversion model can also be a model based on a machine learning network framework.

[0113] For example, the target image style conversion model can be a model based on one of the CNN, DNN, GAN and other machine learning networks.

[0114] In an embodiment, the basic image style conversion model can realize the conversion of the image style because the developer has trained the preset basic image style conversion model before using the basic image style model, so that the trained basic image style conversion model masters the rules of style conversion and can convert the image style based on the rules of style conversion.

[0115] The preset basic image style conversion model includes a model that is pre-set and not trained. For example, the preset basic image style conversion model can be a GAN with only a network architecture, but the GAN does not have any function and cannot realize the conversion of the image to the basic image style.

[0116] The training of the model can include a process of enabling the model to learn from massive data, so that the model can summarize the rules from the massive data and can process any input data in the model based on the rules.

[0117] Therefore, asFigure 4 As shown, before obtaining the basic image style conversion model, the preset basic image style conversion model can be trained, so as to obtain the basic image style conversion model. For example, the preset basic image style conversion model can be trained by using the basic image style features and the training images, so as to obtain the basic image style conversion.

[0118] Wherein, since the function of the target image style conversion model is similar to that of the basic image style conversion model, both are for realizing image style conversion, only the target image style conversion model can be for some specific styles. Therefore, the preset target image style conversion model can be trained by using the basic image style conversion model and the target image style features, so as to improve the efficiency of training the preset target image style conversion model.

[0119] In addition, in an embodiment, according to the training process of the model, the model based on the machine learning network framework can be divided into a model based on supervised learning and a model based on unsupervised learning.

[0120] Wherein, supervised learning can include adjusting the parameters in the model by using a set of known category samples, so that the model reaches the required performance. Supervised learning is a machine learning task of inferring a function from labeled training data. The training data includes a set of training examples. In supervised learning, each example is composed of an input object (usually a vector) and an expected output value (also called a supervision signal). The supervised learning algorithm analyzes the training data and produces an inferred function that can be used to map new examples.

[0121] However, in real life, there are often problems such as lack of sufficient prior knowledge, so it is difficult to manually annotate categories or the cost of manual category annotation is too high. Naturally, we hope that the model can replace the developer to complete these tasks, or at least provide some help. The process of training the model according to the training samples whose categories are unknown (not labeled) is called unsupervised learning.

[0122] Wherein, the training set includes the data for training the model. For example, when training the preset target image style conversion model, the preset target image style conversion model is trained by using the target image style features, so the training set is the target image style features.

[0123] In an embodiment, according to the model framework of the preset target image style conversion model, the training set, etc., it can be determined whether to use supervised learning or unsupervised learning to train the target image style conversion model.

[0124] For example, when the model framework of the preset target image style conversion model is CNN, and the training set has label information, the preset target image style conversion model can be trained in a supervised learning manner.

[0125] For another example, when the model framework of the preset target image style conversion model is GAN, but the training set is relatively rich, the model framework of the preset target image style conversion model can be trained in an unsupervised learning manner.

[0126] In an embodiment, when the model is trained in an unsupervised learning manner, the quantity of the training set needs to be compared with the style and sufficient, so that the model can learn from a large amount of data and summarize the rules for solving the problem.

[0127] However, since the target style image is generally small in quantity, if the target image style conversion model to be trained is directly trained by using the target style image, the effect of the target image style conversion model will be poor. Therefore, as shown in Figure 4 the better-performing basic image style conversion model can be used to perform transfer training on the target image style conversion model to be trained.

[0128] Specifically, the step of "performing transfer training on the preset target image style conversion model by using the basic image style conversion model and the target image style feature to obtain a target image style conversion model" can include:

[0129] initializing the model parameters of the preset target image style conversion model by using the model parameters of the basic image style conversion model to obtain an initialized target image style conversion model;

[0130] training the initialized target image style conversion model by using the target image style feature to obtain the target image style conversion model.

[0131] The preset target image style conversion model includes a model that is pre-set and not trained. For example, the preset target image style conversion model can be a GAN with only a network architecture, but the GAN does not have any function and cannot realize the model of converting an image into a target image style.

[0132] In an embodiment, in order to enable the basic image style conversion model to perform transfer training on the preset target image style conversion model, the model architecture of the basic image style conversion model is generally the same as that of the preset target image style conversion model.

[0133] For example, the model architecture of the basic image style conversion model includes three functional layers, and the model architecture of the preset target image style conversion model also includes three functional layers.

[0134] In an embodiment, when the model parameters of the target image style conversion model to be trained are initialized by using the model parameters of the base image style conversion model, the model parameters of the preset target image style conversion model can be set according to the model parameters of the base image style conversion model, so that the initialized target image style conversion model has the basic image style conversion capability.

[0135] For example, the parameters of the three functional layers in the base image style conversion model are k1, k2 and k3, and the parameters of the three functional layers in the preset target image style conversion model can also be k1, k2 and k3.

[0136] In an embodiment, in order to enable the target image style conversion model to convert the style of an image into a target image style, the target image style conversion model can be trained by using the target style image, so as to obtain the target image style conversion model.

[0137] Specifically, the step of "training the initialized target image style conversion model by using the target image style feature to obtain the target image style conversion model" can include:

[0138] obtaining a training image;

[0139] performing style conversion processing on the training image by using the target image style feature and the initialized target image style conversion model to obtain a style-converted image;

[0140] calculating loss information of the style-converted image and the preset target style image;

[0141] adjusting the model parameters of the initialized target image style conversion model according to the loss information to obtain the base image style conversion model.

[0142] The training image can include an image converted by using the base image style conversion model to be trained. For example, the training image can be a real person selfie, etc. For example, the base image style conversion model to be trained can include a model that can convert a real person selfie into an animation style photo, etc.

[0143] In an embodiment, the process of training the initialized target image style conversion model can be a process of continuously learning to perform style conversion processing on the training image by using the initialized target image style conversion model. Specifically, the step of "performing style conversion processing on the training image by using the target image style feature and the initialized target image style conversion model to obtain a style-converted image" can include:

[0144] The initialization target image style conversion model is used to encode the training image to obtain a training image style feature;

[0145] The target image style feature is used to perform feature conversion processing on the training image style feature to obtain a style-converted feature of the training image.

[0146] The initialization target image style conversion model is used to generate a style-converted image of the training image according to the style-converted feature.

[0147] In an embodiment, when the initialization target image style conversion model is a GAN, the initialization target image style conversion model can be composed of a generator network and a discriminator network. The generator network generates content, and the discriminator network discriminates the content generated by the generator network, so that the content generated by the generator network is more vivid and real. The core logic of the GAN is a process in which the generator network and the discriminator network are in mutual opposition and mutual game.

[0148] For example, the generator network can convert the style of an image to obtain a style-converted image. Then, the discriminator network can discriminate the style-converted image. If the style of the style-converted image meets the discrimination condition, the style-converted image can pass the discrimination of the discriminator network. If the style of the style-converted image does not meet the discrimination condition, the style-converted image cannot pass the discrimination of the discriminator network, and the generator network generates an image with a more vivid style at this time.

[0149] The generator network can be composed of a decoder and an encoder. Both the decoder and the encoder can be a machine learning network. For example, both the decoder and the encoder can be a CNN network.

[0150] In an embodiment, when the initialization target image style conversion model is used to encode the training image, the encoder can be used to encode the training image to obtain the training image style feature.

[0151] In an embodiment, when the initialization target image style conversion model is used to encode the training image, the initialization target image style conversion model can be used to extract features of the training image to obtain feature information of the training image. Then, the training image style feature is obtained according to the feature information of the training image. Specifically, the step of “using the initialization target image style conversion model to encode the training image to obtain a training image style feature” can include:

[0152] The initialization target image style conversion model is used to extract features of the training image to obtain feature information of the training image;

[0153] The feature information is subjected to distribution mapping processing by using the initialized target image style conversion model to obtain distribution features of the feature information.

[0154] The training image style features of the training image are generated according to the distribution features by using the initialized target image style conversion model.

[0155] The feature information of the training image includes information that can represent image features of the training image.

[0156] In an embodiment, the focus of information extraction is different when feature extraction is performed on the training image according to different contents of the training image.

[0157] For example, when the content of the training image is a selfie of a human face, the focus of feature extraction of the training image is feature information of the human face. For another example, when the content of the training image is a landscape, the focus of feature extraction of the training image is global features of the image. For another example, when the content of the training image is a food photo, the focus of feature extraction of the training image is feature information of the food.

[0158] In an embodiment, the training image can be subjected to feature extraction in multiple ways. For example, the image can be subjected to convolution operation by using a gradient operator of the to-be-trained basic image style conversion model to obtain the feature information of the training image. For another example, the image can be subjected to convolution operation by using a convolution kernel of the to-be-trained basic image style conversion model to obtain the feature information of the training image, and the like.

[0159] In an embodiment, after the feature information of the training image is obtained, the feature information can be subjected to distribution mapping processing to obtain distribution features of the feature information.

[0160] The distribution features can include features of the feature information of the training image in a preset distribution rule. The preset distribution rule can include a pre-set image distribution rule. For example, the preset distribution rule can include normal distribution, chi-square distribution, exponential distribution, Poisson distribution, and the like.

[0161] For example, when the preset distribution rule is chi-square distribution, the distribution features can be distribution probability values of the feature information of the training image in the chi-square distribution.

[0162] When the feature information is subjected to distribution mapping processing, the feature information can be subjected to distribution mapping processing by using the preset distribution rule. For example, the distribution rule of the normal distribution is as follows:

[0163]

[0164] According to the distribution rule, a distribution feature of the characteristic information can be obtained. The x can be the characteristic information of the training image. The f(x) can be the distribution feature of the training image. The distribution feature can be a vector or a matrix, etc.

[0165] In an embodiment, the training image style feature of the training image can be generated according to the distribution feature by using the initialized target image style conversion model.

[0166] For example, the distribution feature can be converted into the training image style feature by using a preset latent vector. The preset latent vector includes a vector that is previously set in the initialized target image style conversion model, and the vector can convert the distribution feature into the training image style feature.

[0167] In an embodiment, after obtaining the style feature of the training image, the training image style feature can be converted by using the target style feature, so as to obtain the converted style feature of the training image.

[0168] When the training image is converted by using the target style feature, a plurality of ways can be used.

[0169] For example, the target style feature and the training image style feature can be added, so as to obtain the converted style feature of the training image.

[0170] For another example, the target style feature and the training image style feature can be subjected to adaptive instance normalization (AdaIN), so as to obtain the converted style feature of the training image.

[0171] The AdaIN is a method that can align the mean and variance of the image feature to the mean and variance of the style image, so as to realize the style conversion of the image. For example, the AdaIN can align the mean and variance of the training image style feature to the mean and variance of the basic style feature, so as to obtain the converted style feature.

[0172] When the target style feature and the training image style feature are subjected to the adaptive instance normalization, the following formula can be used:

[0173]

[0174] The x can represent the training image style feature, and the y can represent the target style feature. The mean and the standard deviation can represent the mean and the standard deviation, respectively. The above formula illustrates how to align the mean and the variance of the training image style feature to the mean and the variance of the target style feature by using the AdaIN. and The mean and the standard deviation can represent the mean and the standard deviation, respectively. The above formula illustrates how to align the mean and the variance of the training image style feature to the mean and the variance of the target style feature by using the AdaIN.

[0175] In an embodiment, after the style-transformed features are obtained, an initialized target image style transformation model can be utilized to generate a style-transformed image of the training image according to the style-transformed features. Specifically, the step of “utilizing the initialized target image style transformation model to generate a style-transformed image of the training image according to the style-transformed features” can include:

[0176] decoding the style-transformed features by utilizing the initialized target image style transformation model to obtain a decoded image;

[0177] performing semantic segmentation on the training image to obtain semantic information of the training image;

[0178] performing feature enhancement on the decoded image by utilizing the semantic information to obtain the style-transformed image.

[0179] In an embodiment, when the preset target image style transformation model is a GAN, a decoder in the generation network can be utilized to decode the style-transformed features to obtain a decoded image.

[0180] The decoded image includes an image that already has the target image style features. However, because the contents of different training images are different, the decoded image can be subjected to feature enhancement according to the content of the training image, so that the obtained style-transformed image is more vivid.

[0181] For example, when the content of the training image is a selfie of a human face, the feature of the human face in the decoded image can be enhanced, so that the obtained style-transformed image is more vivid. For another example, when the training image is a landscape image, the global feature of the decoded image can be enhanced, so that the obtained style-transformed image is more vivid.

[0182] In an embodiment, the training image can be subjected to semantic segmentation to obtain semantic information of the training image. Then, the semantic information can be utilized to perform feature enhancement on the decoded image to obtain the style-transformed image.

[0183] The semantic segmentation can refer to classifying each piece of information in a scene to identify information of a certain type.

[0184] For example, when the content of the training image is a selfie of a human face, semantic segmentation can be used to distinguish the information of the human face and the background information. In addition, the information of the five features of the human face can also be distinguished. For example, the location of the eyes and the nose of the human face can be distinguished.

[0185] Semantic information includes information that describes the content of the training image. For example, when the training image is a selfie, semantic information can reveal the location of the face, the background, and the facial features. Semantic information can also be a semantic segmentation mask. This mask can be a digital description of the semantic information, allowing us to determine the location of the face in the training image.

[0186] In one embodiment, various methods can be used to perform semantic segmentation on training images. For example, two-dimensional (2D) semantic segmentation or three-dimensional (3D) semantic segmentation can be used to perform semantic segmentation on point clouds. As another example, machine learning networks such as PointNet and PointNet++ can be used to perform semantic segmentation on training images.

[0187] In one embodiment, when using semantic information to perform feature enhancement processing on the encoded image, the semantic information and the feature information of the encoded image can be superimposed to achieve feature enhancement processing on the encoded image.

[0188] For example, when the training image is a selfie and the encoded image is an anime-style selfie, the facial semantic information of the training image can be superimposed on the feature information of the encoded image to enhance the facial features of the encoded image, making the style-transformed image more vivid and distinctive.

[0189] In one embodiment, after obtaining the style-transformed image, loss information between the style-transformed image and the preset target style image can be calculated, thereby adjusting the model parameters of the preset target image style transformation model based on the loss information to obtain the target image style transformation model.

[0190] The loss information includes metrics that can be used to evaluate the style similarity between the style-transferred image and the target style image. For example, the loss information can be a single value; a smaller value indicates a higher style similarity between the style-transferred image and the target style image, and thus better model performance. Conversely, a larger value indicates a lower style similarity between the style-transferred image and the target style image, and thus worse model performance.

[0191] In one embodiment, a loss function can be used to calculate the loss information between the style-transformed image and the preset target style image.

[0192] The loss function may include loss function Loss functions, 0-1 loss functions, cross-entropy loss functions, exponential loss functions, and so on.

[0193] In an embodiment, when the preset target image style conversion model is a GAN, the loss information of the image after style conversion and the preset target style image can also be calculated by using the discriminator network in the GAN.

[0194] In an embodiment, after obtaining the loss information, the model parameters of the preset target image style conversion model can be adjusted according to the loss information, so as to obtain the target image style conversion model.

[0195] For example, when the loss information is large, the model parameters of the preset target image style conversion model can be adjusted. Then, the adjusted target image style conversion model is trained to see whether the loss information is improved. The above training is repeated until the loss information meets the requirements, and the current adjusted target image style conversion model is determined as the target image style conversion model.

[0196] In an embodiment, when the preset base image style conversion model is trained, since the base image style can include the styles of multiple images, the base image style feature can include the style features of multiple images, so the training set of the preset base image style conversion model is generally rich and sufficient. Therefore, the preset base image style conversion model can be trained by using the base image style feature, so as to obtain the base image style conversion model. The process of training the preset base image style conversion model by using the base style image can refer to the process of training the initialized target image style conversion model, which is not repeated here.

[0197] In the embodiments of the present application, the base image style conversion model to be trained is trained, so that the base image style conversion model can learn the features of multiple styles and has the basic ability of style conversion of images. Then, the target image style conversion model to be trained is trained by using the base image style feature, so as to improve the efficiency of model training.

[0198] In the process of training the model, the performance of the model can be further improved by using the semantic information of the training image to enhance the image, so that the image after style conversion by the model is close to the target image style and has distinct features.

[0199] 104、Based on the model parameters of the base image style conversion model, the model parameters of the target image style conversion model are adjusted to obtain an adjusted target image style conversion model.

[0200] In an embodiment, in order to further improve the performance of the target image style conversion model, the target image style conversion model can also be adjusted by using a basic image style conversion model with stronger generalization ability, so as to obtain an adjusted target image style conversion model. The adjusted target image style conversion model has better performance, and the image converted by the adjusted target image style conversion model can be closer to the target image style.

[0201] In an embodiment, the step of "adjusting the model parameters of the target image style conversion model based on the model parameters of the basic image style conversion model to obtain an adjusted target image style conversion model" can include:

[0202] selecting at least one to-be-adjusted functional layer from the target image style conversion model;

[0203] selecting a corresponding adjustment reference layer in the basic image style conversion model according to the to-be-adjusted functional layer;

[0204] adjusting the parameters of the to-be-adjusted functional layer based on the parameters of the adjustment reference layer to obtain an adjusted target image style conversion model.

[0205] In an embodiment, the network structure of the machine learning network can be composed of a plurality of functional layers, each of which has its function. For example, the network structure of the CNN network can include a feature extraction layer, a pooling layer, a full connection layer, etc. Each functional layer is composed of corresponding calculation factors. For example, the feature extraction layer can be composed of a convolution kernel, the pooling layer can be composed of a pooling function, etc. Therefore, the model parameters can refer to the parameters in the calculation factors in the functional layer. For example, the model parameters can refer to the function coefficients of the pooling function in the pooling layer, etc.

[0206] In an embodiment, when adjusting the model parameters of the target image style conversion model based on the model parameters of the basic image style conversion model, at least one to-be-adjusted functional layer can be first selected from the target image style conversion model. The to-be-adjusted functional layer includes a functional layer whose performance needs to be adjusted. For example, when the performance of the feature extraction layer in the target image style conversion model is poor, the feature extraction layer can be determined as the to-be-adjusted functional layer and selected out. For another example, when the performance of both the feature extraction layer and the pooling layer in the target image style conversion model is poor, both the feature extraction layer and the pooling layer can be extracted and determined as the to-be-adjusted functional layer.

[0207] In an embodiment, the corresponding adjustment reference layer can be selected in the basic image style conversion model according to the to-be-adjusted functional layer. The adjustment reference layer includes a functional layer that is referenced when adjusting the to-be-adjusted functional layer.

[0208] For example, when the performance of the feature extraction layer and the pooling layer in the target image style conversion model is poor, the feature extraction layer and the pooling layer in the basic image style conversion model can be screened accordingly, and the feature extraction layer and the pooling layer in the basic image style conversion model are determined as the adjustment reference layer.

[0209] In an embodiment, after the function layer to be adjusted and the adjustment reference layer are screened, the parameters of the adjustment reference layer can be adjusted to the parameters of the function layer to be adjusted to obtain an adjusted target image style conversion model. When the parameters of the adjustment reference layer are adjusted to the parameters of the function layer to be adjusted, the parameters of the adjustment reference layer and the parameters of the function layer to be adjusted can be fused to obtain the adjusted target image style conversion model. Specifically, the step of "adjusting the parameters of the function layer to be adjusted to the parameters of the adjustment reference layer to obtain an adjusted target image style conversion model" can include:

[0210] determining the parameter fusion strength of the function layer to be adjusted and the parameter fusion strength of the adjustment reference layer;

[0211] performing parameter fusion processing on the parameters of the function layer to be adjusted and the parameters of the adjustment reference layer according to the parameter fusion strength of the function layer to be adjusted and the parameter fusion strength of the adjustment reference layer to obtain an adjusted target image style conversion model.

[0212] For example, the parameters of the function layer to be adjusted are , and the parameters of the adjustment reference layer are . The parameter fusion strength of the function layer to be adjusted is , and the parameter fusion strength of the adjustment reference layer is . When the parameters of the function layer to be adjusted and the parameters of the adjustment reference layer are fused, the parameters and the parameter fusion strength are multiplied and then added. For example, it can be as follows:

[0213]

[0214] Where C is the new fused parameter. Through parameter fusion, the parameters of the target image style conversion model can be changed so that the adjusted target image style conversion model can not only convert the style of an image to a target image style, but also improve the ability to convert the style of an image through the basic image style conversion model.

[0215] 105、obtain the basic image style feature and the image to be converted, and perform feature fusion processing on the basic image style feature and the target image style feature to obtain a fused image style feature.

[0216] In an embodiment, the image to be converted can include a selfie. Wherein, the image to be converted can be any image, such as a selfie, a landscape, or an animal photo, etc.

[0217] For example, when the image to be converted is a selfie, the target image style can be an animation style. That is, the image style conversion method proposed in the embodiments of the present application can convert a selfie into an animation style photo.

[0218] For another example, when the image to be converted is a landscape, the target image style can be an oil painting style. That is, the image style conversion method proposed in the embodiments of the present application can convert a landscape into an oil painting style photo. In an embodiment, the basic image style features include information that can best reflect the characteristics of the image style.

[0219] Wherein, since various different image styles can be collectively referred to as a basic image style, the basic image style features can also include features of various different image styles. For example, if the basic image style includes 10 different image styles, the basic image style features can include 10 different image style features. For example, if the basic image style includes 10 different animation styles, and each animation style has the painting style of an animation artist, the basic image style features can also include 10 different animation style features.

[0220] In an embodiment, the basic image style features and the target image style features can also be subjected to feature fusion processing to obtain fused image style features.

[0221] Wherein, since the basic image style features can include multiple image style features, the statistical processing can be performed and then fused with the target image style features. Specifically, the step of “performing feature fusion processing on the basic image style features and the target image style features to obtain fused image style features” can include:

[0222] statistically processing the basic image style features to obtain statistical image style features;

[0223] fusing the statistical image style features and the target image style features to obtain fused image style features.

[0224] Wherein, the basic image style features can be statistically processed in various ways. For example, multiple image style features can be averaged to obtain statistical image style features. For another example, multiple image style features can be subjected to variance calculation to obtain statistical image style features.

[0225] After the image style features are obtained, the image style features and the target image style features can be fused to obtain fused image style features.

[0226] For example, the image style features and the target image style features can be added to obtain the fused image style features.

[0227] In an embodiment, the step of “fusing the base image style features and the target image style features to obtain fused image style features” and the step of “adjusting the model parameters of the target image style conversion model using the model parameters of the base image style conversion model to obtain an adjusted target image style conversion model” are not limited in execution order. For example, the step of “fusing the base image style features and the target image style features to obtain fused image style features” can be performed first, or the step of “adjusting the model parameters of the target image style conversion model using the model parameters of the base image style conversion model to obtain an adjusted target image style conversion model” can be performed first. For another example, the two steps can be performed in parallel.

[0228] 106. Perform style conversion on the to-be-converted image using the adjusted target image style conversion model and the fused image style features to obtain a target image conforming to the target image style.

[0229] In an embodiment, the to-be-converted image can be converted using the adjusted target image style conversion model and the fused image style features to obtain a target image conforming to the target image style. Specifically, the step of “performing style conversion on the to-be-converted image using the adjusted target image style conversion model and the fused image style features to obtain a target image conforming to the target image style” can include:

[0230] Encode the to-be-converted image using the adjusted target image style conversion model to obtain to-be-converted style features of the to-be-converted image;

[0231] Convert the to-be-converted style features using the fused image style features to obtain converted style features;

[0232] Decode the converted style features using the adjusted target image style conversion model to obtain a target image conforming to the target image style.

[0233] In an embodiment, when the adjusted target image style conversion model is a GAN, the to-be-converted image can be encoded using an encoder in the generative network to obtain to-be-converted image style of the to-be-converted image.

[0234] In an embodiment, the feature information of the to-be-converted image can be obtained by performing feature extraction on the to-be-converted image by using the adjusted target image style conversion model. Then, the to-be-converted style feature of the to-be-converted image can be obtained according to the feature information of the to-be-converted image. Specifically, the step of "performing encoding processing on the to-be-converted image by using the adjusted target image style conversion model to obtain the to-be-converted style feature of the to-be-converted image" can include:

[0235] performing feature extraction on the to-be-converted image by using the adjusted target image style conversion model to obtain the feature information of the to-be-converted image;

[0236] performing distribution mapping processing on the feature information by using the adjusted target image style conversion model to obtain the distribution feature of the feature information;

[0237] generating the to-be-converted style feature of the to-be-converted image according to the distribution feature by using the adjusted target image style conversion model.

[0238] In an embodiment, the feature extraction on the to-be-converted image can be performed in various ways. For example, the gradient operator of the adjusted target image style conversion model can be used to perform convolution operation on the to-be-converted image, so as to obtain the feature information of the to-be-converted image. For another example, the convolution kernel of the adjusted target image style conversion model can be used to perform convolution operation on the to-be-converted image, so as to obtain the feature information of the to-be-converted image, and so on.

[0239] In an embodiment, the distribution mapping processing on the feature information can be performed by using a preset distribution rule.

[0240] In an embodiment, the distribution feature can be converted into the to-be-converted style feature by using a preset latent vector when the distribution feature is generated into the to-be-converted style feature of the to-be-converted image.

[0241] In an embodiment, the to-be-converted style feature can be converted into the converted style feature by using the fused image style feature in various ways.

[0242] For example, the fused image style feature and the to-be-converted style feature can be added to obtain the converted style feature.

[0243] For another example, the fused image style feature and the to-be-converted style feature can be subjected to AdaIN to obtain the converted style feature.

[0244] In an embodiment, when the adjusted target image style conversion model is a GAN, the to-be-converted image can be encoded by using the decoder in the generation network to obtain a target image conforming to the target image style.

[0245] In an embodiment, after obtaining the target image, the quality of the target image can be detected, and when the quality of the target image is not good enough, the quality of the target image can be improved. Specifically, the method proposed in the embodiments of the present application can further include:

[0246] performing identification processing on the target image to obtain quality information of the target image;

[0247] performing quality adjustment processing on the target image according to the quality information to obtain an adjusted target image.

[0248] The quality information of the target image includes information that can represent the quality of the image. For example, the quality information can include the resolution, image size, and the like of the target image.

[0249] In an embodiment, the quality adjustment processing can be performed on the target image according to the quality information to obtain an adjusted image. For example, when the resolution of the target image is too low, the resolution of the target image can be improved. For another example, when the quality of the target image is poor due to the image size being too small or the like, the quality of the image can be improved by adjusting the size of the image and adding pixel information to the image.

[0250] In an embodiment, the target image can also be adjusted by using an image adjustment model. The image adjustment model can be any machine learning model. For example, the image adjustment model can be a CNN model or a DNN model, and the like. For another example, the structure of the image adjustment model can be composed of a convolution layer (Conv), a batch normalization (BN) layer, and a rectified linear unit (ReLU) function.

[0251] The Conv layer can include a plurality of convolution kernels, and each convolution kernel can sample the image to obtain feature information of the image.

[0252] The BN layer can normalize the information output by the Conv layer to obtain normalized information.

[0253] The ReLU function is a nonlinear function that can perform nonlinear transformation on information.

[0254] In an embodiment, before the target image is adjusted by using the image adjustment model, a preset image adjustment model can be obtained, and the preset image adjustment model can be trained to obtain the image adjustment model. The steps of training the preset image adjustment model can include:

[0255] obtaining an adjustment reference image and a to-be-trained image adjustment model;

[0256] degrading the adjustment reference image to obtain a degraded image;

[0257] training the preset image adjustment model using the degraded image to obtain the image adjustment model.

[0258] The adjustment reference image can include high-quality images of any style. For example, the adjustment reference image can include high-definition and noise-free animation images of any style.

[0259] The preset image adjustment model can be a model stacked with Conv-BN-Relu as the network structure.

[0260] In an embodiment, since the adjustment reference image is usually small in quantity, the adjustment reference image can be degraded to obtain a degraded image. Then, the degraded image is used to train the to-be-trained image adjustment model, so as to obtain the image adjustment model. For example, as shown in FIG. 6, the adjustment reference image can be randomly degraded to obtain a degraded image. Then, the degraded image can be used to train the preset image adjustment model, so as to obtain the image adjustment model. Figure 5

[0261] The degradation processing includes processing for reducing the quality of the adjustment reference image. The degradation processing can have various modes. For example, random noise can be added to the adjustment reference image to reduce the quality of the adjustment reference image. For another example, the image size of the adjustment reference image can be randomly scaled to reduce the quality of the adjustment reference image. For another example, the pixels of the adjustment reference image can be randomly averaged to reduce the quality of the adjustment reference image.

[0262] After obtaining the degraded image, the degraded image can be used to train the to-be-trained image adjustment model, so as to obtain the image adjustment model.

[0263] In an embodiment, after obtaining the image adjustment model, the image adjustment model can be used to perform quality adjustment processing on the target image. For example, as shown in FIG. 7, if the target image has a problem of low resolution, the image adjustment model can be used to perform quality adjustment processing on the target image, so as to improve the quality of the target image. Figure 6

[0264] ​​The embodiment of the present application provides an image style conversion method, and the image style conversion method comprises the following steps: acquiring a basic image style conversion model, a target image style conversion model, a basic image style feature, a target image style feature and a to-be-converted image, wherein the basic image style conversion model is used for converting the image style into a basic image style, and the target image style conversion model is used for converting the image style into a target image style; adjusting model parameters of the target image style conversion model by using model parameters of the basic image style conversion model, to obtain an adjusted target image style conversion model; performing feature fusion processing on the basic image style feature and the target image style feature, to obtain a fused image style feature; and performing style conversion processing on the to-be-converted image by using the adjusted target image style conversion model and the fused image style feature, to obtain a target image conforming to the target image style. By adjusting the model parameters of the target image style conversion model by using the model parameters of the basic image style conversion model, the target image generated by the adjusted target image style conversion model is closer to the target image style, and the accuracy of the image style conversion is improved.

[0265] Moreover, the fused image style feature is also used when the target image is generated. Since the fused image style feature is obtained by performing feature fusion on the basic image style feature and the target image style feature, the fused image style feature not only has pertinence, but also has generalization, and the accuracy of the image style conversion can be further improved.

[0266] In addition, the embodiment of the present application can also improve the quality of the target image, and the quality of the target image is improved.

[0267] According to the method described in the above embodiment, the following will be further described in detail by way of example.

[0268] The embodiment of the present application will take the image style conversion method integrated on the computer device as an example to introduce the method of the embodiment of the present application.

[0269] The image style conversion method can be the style conversion of the image. For example, a real object image can be converted into a comic style image. For example, as shown in FIG. 1, a person image can be converted into a comic style image. Next, the image style conversion method proposed by the embodiment of the present application will be further described in detail in the application scenario. Figure 7

[0270] In an embodiment, as shown in FIG. 2, an image style conversion method is provided, and the specific process is as follows: Figure 8

[0271] ​​201、The computer device obtains a target style image, a preset basic image style conversion model, a preset target image style conversion model, basic image style features, target image style features, and an image to be converted.

[0272] The target style image can include a certain target style of a comic image.

[0273] The preset basic image style conversion model can be a GAN model. The preset target image style conversion model can also be a GAN model.

[0274] For example, as shown in the following figure, the GAN model (i.e., 009 in the following figure) can include a generation network and a discrimination network. The generation network can include an encoder and a decoder. Figure 9 Figure 6 The basic image style features can include features of a sufficient amount of and easily obtained animation style image.

[0275] In an embodiment, before obtaining the basic image style features, a style encoder can be used to encode images of various different styles, thereby obtaining features of various different image styles.

[0276] For example, as shown in the following figure, a style encoder can be used to encode images of various animation styles, thereby obtaining the basic image style features. The basic image style features can be represented as

[0277] Figure 9 The target image style features can include features of a certain target style of a comic image.

[0278] The image to be converted can include an image that needs to be converted in image style. For example, the image to be converted can be a non-comic image, etc.

[0279] The image to be converted can include an image that needs to be converted in image style. For example, the image to be converted can be a non-comic image, etc.

[0280] 202、The computer device trains the preset basic image style conversion model using the basic image style features, thereby obtaining a basic image style conversion model.

[0281] The process of training the preset basic image style conversion model can be as shown in the following figure.

[0282] For example, as shown in the following figure, the computer device can first obtain training image style features by encoding a training image 010 using an encoder. The training image style features can be represented as

[0283] Then, the computer device can perform feature conversion processing on the training image style features using the basic style features, thereby obtaining style-converted features of the training image. For example, as shown in the following figure, the computer device can first obtain training image style features 011 by encoding a training image 010 using an encoder.​​​​Figure 6 As shown, the style converted feature of the training image can be obtained by performing AdaIN on the style feature and the content feature of the training image.

[0284] Next, the computer device can encode the decoded image using the encoder to obtain an encoded image of the training image.

[0285] The computer device can further perform semantic segmentation processing on the encoded image to obtain semantic information of the training image, and then perform feature enhancement processing on the decoded image using the semantic information to obtain the style converted image.

[0286] After obtaining the style converted image, the computer device can use the discriminator network to discriminate the style converted image, so as to calculate the loss information of the style converted image and the preset base style image.

[0287] Then, the computer device can adjust the model parameters of the generator using the loss information, so as to improve the style conversion capability of the generator, so as to obtain a base image style conversion model with required performance.

[0288] 203、The computer device initializes the model parameters of the preset target image style conversion model using the model parameters of the base image style conversion model to obtain an initialized target image style conversion model.

[0289] 204、The computer device trains the initialized target image style conversion model using the target style image to obtain a target image style conversion model.

[0290] The computer device trains the initialized target image style conversion model using the target style image in the manner described above, which will not be repeated here.

[0291] 205、The computer device adjusts the model parameters of the base image style conversion model and the model parameters of the target image style conversion model to obtain an adjusted target image style conversion model.

[0292] For example, the computer device selects at least one to-be-adjusted functional layer from the target image style conversion model. Then, the computer device can select a corresponding adjustment reference layer in the base image style conversion model according to the to-be-adjusted functional layer. Next, the computer device can adjust the parameters of the to-be-adjusted functional layer using the parameters of the adjustment reference layer to obtain the adjusted target image style conversion model.

[0293] ​​​​In the parameter adjustment process of the to-be-adjusted functional layer using the parameters of the adjustment reference layer, the computer device can determine the parameter fusion strength of the to-be-adjusted functional layer and the parameter fusion strength of the adjustment reference layer. Then, the computer device can perform parameter fusion processing on the parameters of the to-be-adjusted functional layer and the parameters of the adjustment reference layer according to the parameter fusion strength of the to-be-adjusted functional layer and the parameter fusion strength of the adjustment reference layer, to obtain the adjusted target image style conversion model.

[0294] For example, the parameter of the to-be-adjusted functional layer is a1, and the parameter of the adjustment reference layer is b1. The parameter fusion strength of the to-be-adjusted functional layer is p1, and the parameter fusion strength of the adjustment reference layer is q1. When performing parameter fusion processing on the parameters of the to-be-adjusted functional layer and the parameters of the adjustment reference layer, the parameters and the parameter fusion strengths are multiplied and then added. For example, the following can be used:

[0295]

[0296] Wherein C is the new parameter after fusion.

[0297] 206, the computer device performs feature fusion processing on the basic image style feature and the target image style feature to obtain a fused image style feature.

[0298] For example, the computer device performs statistical processing on the basic image style feature to obtain a statistical image style feature. Then, the computer device performs fusion processing on the statistical image style feature and the target image style feature to obtain a fused image style feature.

[0299] For example, a plurality of image style features can be averaged to obtain a statistical image style feature. For another example, a plurality of image style features can be squared to obtain a statistical image style feature.

[0300] 207, the computer device performs style conversion processing on the to-be-converted image using the adjusted target image style conversion model and the fused image style feature, to obtain a target image conforming to the target image style.

[0301] For example, as shown in Figure 10 the non-comic image can be converted into a comic style image through the adjusted target image style conversion model. In this embodiment of the present application, the parameters of the adjusted target image style conversion model can be adjusted to generate images of different comic styles. For example, by adjusting the parameters of the adjusted target image style conversion model, an image as shown in comic style 011 can be generated, or an image as shown in comic style 012 can be generated, or an image as shown in comic style 013 can be generated.

[0302] In the embodiment of the present application, the computer device can obtain a base image style conversion model, a target image style conversion model, a base image style feature, a target image style feature, and a to-be-converted image; the computer device adjusts the model parameters of the target image style conversion model by using the model parameters of the base image style conversion model, to obtain an adjusted target image style conversion model; the computer device performs feature fusion processing on the base image style feature and the target image style feature, to obtain a fused image style feature; and the computer device performs style conversion processing on the to-be-converted image by using the adjusted target image style conversion model and the fused image style feature, to obtain a target image conforming to the target image style, thereby improving the accuracy of style conversion of the image.

[0303] To better implement the image style conversion method provided in the embodiments of the present application, in an embodiment, an image style conversion device is further provided, which can be integrated in a computer device. The meanings of the terms are the same as those in the above image style conversion method, and the specific implementation details can be referred to the description in the method embodiment.

[0304] In an embodiment, an image style conversion device is provided, which can be specifically integrated in a computer device, as shown in Figure 11 The image style conversion device includes an obtaining unit 301, a feature extraction unit 302, a migration training unit 303, an adjusting unit 304, a feature fusion unit 305, and a style conversion unit 306, and the details are as follows.

[0305] The obtaining unit 301 is configured to obtain a to-be-processed target style image, and perform quality adjustment processing on the to-be-processed target style image, to obtain an adjusted target style image.

[0306] The feature extraction unit 302 is configured to perform feature extraction on the adjusted target style image, to obtain a target image style feature.

[0307] The migration training unit 303 is configured to obtain a base image style conversion model and a target image style feature, and perform migration training on a preset target image style conversion model by using the base image style conversion model and the target image style feature, to obtain a target image style conversion model.

[0308] The adjusting unit 304 is configured to adjust the model parameters of the target image style conversion model based on the model parameters of the base image style conversion model, to obtain an adjusted target image style conversion model.

[0309] The feature fusion unit 305 is configured to acquire a basic image style feature and a to-be-converted image, and perform feature fusion processing on the basic image style feature and the target image style feature to obtain a fused image style feature.

[0310] The style conversion unit 306 is configured to perform style conversion processing on the to-be-converted image by using the adjusted target image style conversion model and the fused image style feature to obtain a target image conforming to a target image style.

[0311] In an embodiment, the style conversion unit 306 can include:

[0312] The encoding subunit is configured to perform encoding processing on the to-be-converted image by using the adjusted target image style conversion model to obtain a to-be-converted style feature of the to-be-converted image.

[0313] The feature conversion subunit is configured to perform feature conversion processing on the to-be-converted style feature by using the fused image style feature to obtain a converted style feature.

[0314] The decoding subunit is configured to perform decoding processing on the converted style feature by using the adjusted target image style conversion model to obtain the target image conforming to the target image style.

[0315] In an embodiment, the encoding subunit can include:

[0316] The feature extraction module is configured to perform feature extraction processing on the to-be-converted image by using the adjusted target image style conversion model to obtain feature information of the to-be-converted image.

[0317] The distribution mapping module is configured to perform distribution mapping processing on the feature information by using the adjusted target image style conversion model to obtain a distribution feature of the feature information.

[0318] The generation module is configured to generate, by using the adjusted target image style conversion model, the to-be-converted style feature of the to-be-converted image according to the distribution feature.

[0319] In an embodiment, the feature fusion unit 305 can include:

[0320] The statistical subunit is configured to perform statistical processing on the basic image style feature to obtain a statistical image style feature.

[0321] The fusion subunit is configured to perform fusion processing on the statistical image style feature and the target image style feature to obtain a fused image style feature.

[0322] In an embodiment, the adjustment unit 304 can include:

[0323] a first screening subunit configured to screen at least one to-be-adjusted functional layer from the target image style conversion model;

[0324] a second screening subunit configured to screen a corresponding adjustment reference layer from the base image style conversion model according to the to-be-adjusted functional layer;

[0325] an adjustment subunit configured to adjust parameters of the to-be-adjusted functional layer by using parameters of the adjustment reference layer to obtain the adjusted target image style conversion model.

[0326] In an embodiment, the adjustment subunit can include:

[0327] a determination module configured to determine parameter fusion intensities of the to-be-adjusted functional layer and the adjustment reference layer;

[0328] a parameter fusion module configured to perform parameter fusion processing on the parameters of the to-be-adjusted functional layer and the parameters of the adjustment reference layer according to the parameter fusion intensities of the to-be-adjusted functional layer and the adjustment reference layer to obtain the adjusted target image style conversion model.

[0329] In an embodiment, the migration training unit 303 can include:

[0330] an initialization subunit configured to initialize model parameters of the preset target image style conversion model by using model parameters of the base image style conversion model to obtain an initialized target image style conversion model;

[0331] a training subunit configured to train the initialized target image style conversion model by using the target image style features to obtain the target image style conversion model.

[0332] In an embodiment, the training subunit can include:

[0333] an acquisition model configured to acquire training images;

[0334] a style conversion module configured to perform style conversion processing on the training images by using the target image style features and the initialized target image style conversion model to obtain style-converted images;

[0335] a calculation module configured to calculate loss information of the style-converted images and preset target style images;

[0336] an adjustment module configured to adjust model parameters of the initialized target image style conversion model according to the loss information to obtain the base image style conversion model.

[0337] In one embodiment, the style conversion module may include:

[0338] The encoding submodule is used to encode the training image using the initialization target image style transfer model to obtain the style features of the training image;

[0339] The feature conversion submodule is used to perform feature conversion processing on the style features of the training image using the style features of the target image, so as to obtain the style-converted features of the training image;

[0340] A generation submodule is used to generate a style-transformed image of the training image based on the style-transformed features using the initial target image style transfer model.

[0341] In one embodiment, the generation submodule can be used to:

[0342] Using the initial target image style transfer model, the style-transferred features are decoded to obtain the decoded image;

[0343] The training images are subjected to semantic segmentation to obtain the semantic information of the training images;

[0344] Using the semantic information, feature enhancement processing is performed on the decoded image to obtain the style-transformed image.

[0345] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0346] The image style conversion device described above can improve the accuracy of image style conversion.

[0347] This application also provides a computer device, which may include a terminal or a server. For example, the computer device may serve as an image style conversion terminal, such as a mobile phone, tablet computer, etc.; or it may serve as a server, such as an image style conversion server. Figure 12 As shown, it illustrates the structural diagram of the terminal involved in the embodiments of this application, specifically:

[0348] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 12The computer device structure shown in the figure is not a limitation of the computer device, and can include more or fewer components than shown, or combine certain components, or arrange different components. Among them:

[0349] The processor 401 is the control center of the computer device, connects various parts of the computer device through various interfaces and lines, and performs various functions of the computer device and processes data by running or executing software programs and / or modules stored in the memory 402 and calling data stored in the memory 402, thereby overall detecting the computer device. Optionally, the processor 401 can include one or more processing cores; preferably, the processor 401 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user pages, and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 401.

[0350] The memory 402 can be used to store software programs and modules, and the processor 401 executes various function applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; the data storage area can store data created according to the use of the computer device, etc. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 402 can also include a memory controller to provide access for the processor 401 to the memory 402.

[0351] The computer device also includes a power supply 403 for powering various components, and preferably the power supply 403 can be logically connected to the processor 401 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 403 can also include one or more direct current or alternating current power supplies, a recharging system, a power failure detection circuit, a power supply converter or inverter, a power supply state indicator, and any other components.

[0352] The computer device can also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0353] Although not shown, the computer device can further include a display unit and the like, which will not be described herein. Specifically in the present embodiment, the processor 401 in the computer device will load the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and run the application program stored in the memory 402 by the processor 401, thereby realizing various functions as follows:

[0354] obtain a to-be-processed target style image, perform quality adjustment processing on the to-be-processed target style image, and obtain an adjusted target style image;

[0355] perform feature extraction on the adjusted target style image, and obtain a target image style feature;

[0356] obtain a basic image style conversion model, perform migration training on a preset target image style conversion model by using the basic image style conversion model and the target image style feature, and obtain a target image style conversion model;

[0357] perform adjustment processing on model parameters of the target image style conversion model based on model parameters of the basic image style conversion model, and obtain an adjusted target image style conversion model;

[0358] obtain a basic image style feature and a to-be-converted image, perform feature fusion processing on the basic image style feature and the target image style feature, and obtain a fused image style feature;

[0359] perform style conversion processing on the to-be-converted image by using the adjusted target image style conversion model and the fused image style feature, and obtain a target image conforming to a target image style.

[0360] The specific implementation of each operation can be referred to the foregoing embodiments, which will not be described herein.

[0361] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional implementation manners in the foregoing embodiments.

[0362] Those skilled in the art can understand that all or part of the steps in the various methods of the foregoing embodiments can be completed by a computer program, or by relevant hardware controlled by a computer program, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0363] To this end, the embodiment of the present application also provides a storage medium, wherein the computer program can be loaded by the processor to execute the steps in any of the image style conversion methods provided by the embodiment of the present application. For example, the computer program can execute the following steps:

[0364] Obtaining a target style image to be processed, performing quality adjustment processing on the target style image to be processed, and obtaining an adjusted target style image;

[0365] Performing feature extraction on the adjusted target style image to obtain a target image style feature;

[0366] Obtaining a basic image style conversion model, performing migration training on a preset target image style conversion model using the basic image style conversion model and the target image style feature, and obtaining a target image style conversion model;

[0367] Adjusting the model parameters of the target image style conversion model based on the model parameters of the basic image style conversion model to obtain an adjusted target image style conversion model;

[0368] Obtaining a basic image style feature and a to-be-converted image, and performing feature fusion processing on the basic image style feature and the target image style feature to obtain a fused image style feature;

[0369] Performing style conversion processing on the to-be-converted image using the adjusted target image style conversion model and the fused image style feature to obtain a target image conforming to a target image style.

[0370] The specific implementation of each operation can be referred to the foregoing embodiments, which will not be described here.

[0371] Since the computer program stored in the storage medium can execute the steps in any of the image style conversion methods provided by the embodiment of the present application, the beneficial effects of any of the image style conversion methods provided by the embodiment of the present application can be achieved, which will be described in detail in the foregoing embodiments, and will not be described here.

[0372] The above describes in detail the image style conversion method, device, computer device and storage medium provided by the embodiment of the present application. The specific examples are applied to describe the principle and implementation manner of the present application. The above embodiment is only used to help understand the method and its core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed. The above description should not be understood as the limitation of the present application.

Claims

1. An image style transfer method, characterized in that, include: Obtain the target style image to be processed, and perform quality adjustment processing on the target style image to obtain the adjusted target style image; Feature extraction is performed on the adjusted target style image to obtain the target image style features; A basic image style transfer model is obtained, and the preset target image style transfer model is transferred and trained using the basic image style transfer model and the target image style features to obtain the target image style transfer model. Based on the model parameters of the basic image style transfer model, the model parameters of the target image style transfer model are adjusted to obtain the adjusted target image style transfer model. Obtain basic image style features and the image to be converted, and perform feature fusion processing on the basic image style features and the target image style features to obtain fused image style features; Using the adjusted target image style transfer model and the fused image style features, the image to be converted is subjected to style transfer processing to obtain a target image that conforms to the style of the target image; The image style includes the style of the image being presented, the target image style includes the style that the user expects the image to be presented, and the image style of the image to be processed is different from the target image style.

2. The image style transfer method as described in claim 1, characterized in that, The step of performing style transfer processing on the image to be converted using the adjusted target image style transfer model and the fused image style features to obtain a target image that conforms to the style of the target image includes: The adjusted target image style transfer model is used to encode the image to be converted to obtain the style features to be converted of the image; The style features to be converted are processed using the style features of the fused image to obtain the converted style features. The adjusted target image style transfer model is used to decode the transferred style features to obtain the target image that conforms to the target image style.

3. The image style transfer method as described in claim 2, characterized in that, The step of encoding the image to be converted using the adjusted target image style transfer model to obtain the style features to be converted of the image includes: Using the adjusted target image style transfer model, feature extraction processing is performed on the image to be converted to obtain the feature information of the image to be converted. Using the adjusted target image style transfer model, the feature information is processed by distribution mapping to obtain the distribution features of the feature information; Using the adjusted target image style transfer model, the style features to be transferred for the image are generated based on the distribution features.

4. The image style transfer method as described in claim 1, characterized in that, The step of performing feature fusion processing on the basic image style features and the target image style features to obtain fused image style features includes: The basic image style features are statistically processed to obtain the statistically derived image style features. The statistically derived image style features and the target image style features are fused together to obtain the fused image style features.

5. The image style transfer method as described in claim 1, characterized in that, The process of adjusting the model parameters of the target image style transfer model based on the model parameters of the basic image style transfer model to obtain the adjusted target image style transfer model includes: Select at least one functional layer to be adjusted from the target image style transfer model; Based on the functional layer to be adjusted, the corresponding adjustment reference layer is selected from the basic image style transfer model; Based on the parameters of the adjustment reference layer, the parameters of the functional layer to be adjusted are processed to obtain the adjusted target image style transfer model.

6. The image style transfer method as described in claim 5, characterized in that, The step of adjusting the parameters of the functional layer to be adjusted based on the parameters of the adjustment reference layer to obtain the adjusted target image style transfer model includes: Determine the parameter fusion strength of the functional layer to be adjusted and the parameter fusion strength of the adjustment reference layer; Based on the parameter fusion strength of the functional layer to be adjusted and the parameter fusion strength of the adjustment reference layer, the parameters of the functional layer to be adjusted and the parameters of the adjustment reference layer are fused to obtain the adjusted target image style transfer model.

7. The image style transfer method as described in claim 1, characterized in that, The step of using the basic image style transfer model and the target image style features to perform transfer training on the preset target image style transfer model to obtain the target image style transfer model includes: Using the model parameters of the basic image style transfer model, the model parameters of the preset target image style transfer model are initialized to obtain the initialized target image style transfer model; The initial target image style transfer model is trained using the target image style features to obtain the target image style transfer model.

8. The image style transfer method as described in claim 7, characterized in that, The step of training the initial target image style transfer model using the target image style features to obtain the target image style transfer model includes: Acquire training images; The style features of the target image and the initialization target image style transfer model are used to perform style transfer processing on the training image to obtain the style-transferred image; Calculate the loss information between the style-transformed image and the preset target style image; The model parameters of the initial target image style transfer model are adjusted based on the loss information to obtain the basic image style transfer model.

9. The image style transfer method as described in claim 8, characterized in that, The step of using the target image style features and the initialized target image style transfer model to perform style transfer processing on the training image to obtain a style-transferred image includes: The training image is encoded using the initial target image style transfer model to obtain the style features of the training image; Using the style features of the target image, feature transformation processing is performed on the style features of the training image to obtain the style-transformed features of the training image; Using the initial target image style transfer model, the style-transferred image of the training image is generated based on the style-transferred features.

10. The image style transfer method as described in claim 9, characterized in that, The step of generating a style-transferred image of the training image based on the style-transferred features using the initialized target image style transfer model includes: Using the initial target image style transfer model, the style-transferred features are decoded to obtain the decoded image; The training images are subjected to semantic segmentation to obtain the semantic information of the training images; Using the semantic information, feature enhancement processing is performed on the decoded image to obtain the style-transformed image.

11. The image style transfer method as described in claim 1, characterized in that, The method further includes: The target image is subjected to recognition processing to obtain the quality information of the target image; Based on the quality information, the target image is subjected to quality adjustment processing to obtain the adjusted target image.

12. The image style transfer method as described in claim 11, characterized in that, The step of performing quality adjustment processing on the target image based on the quality information to obtain the adjusted target image includes: Obtain the image adjustment model; The target image is adjusted using the image adjustment model to obtain the adjusted target image; Before adjusting the target image using an image adjustment model to obtain the adjusted target image, the process includes: Obtain the reference image and preset image adjustment model for adjustment; The adjusted reference image is degraded to obtain a degraded image; The image adjustment model is trained using the degraded image to obtain the image adjustment model.

13. An image style conversion device, characterized in that, include: The acquisition unit is used to acquire the target style image to be processed, perform quality adjustment processing on the target style image to be processed, and obtain the adjusted target style image. The feature extraction unit is used to extract features from the adjusted target style image to obtain target image style features; The transfer training unit is used to acquire a basic image style transfer model and target image style features, and to perform transfer training on a preset target image style transfer model using the basic image style transfer model and the target image style features to obtain the target image style transfer model. The adjustment unit is used to adjust the model parameters of the target image style transfer model based on the model parameters of the basic image style transfer model, so as to obtain the adjusted target image style transfer model. The feature fusion unit is used to acquire basic image style features and the image to be converted, and to perform feature fusion processing on the basic image style features and the target image style features to obtain fused image style features; The style transfer unit is used to perform style transfer processing on the image to be converted using the adjusted target image style transfer model and the fused image style features to obtain a target image that conforms to the style of the target image. The image style includes the style of the image being presented, the target image style includes the style that the user expects the image to be presented, and the image style of the image to be processed is different from the target image style.

14. A computer device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor runs the application program within the memory to perform the operations in the image style transfer method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the image style transfer method according to any one of claims 1 to 12.

Images