Stylized image generation method, device, equipment and storage medium

Abstract

The present invention discloses a method, device, equipment and storage medium for generating a stylized image, the method comprising: determining a target style type and an attribute feature vector corresponding to a target image to be processed, and determining a style strength feature corresponding to the target image according to the attribute feature vector; extracting a global attribute feature of the target image; searching a preset style feature library according to the target style type to obtain a style feature matching the target style type; splicing the global attribute feature, the style feature and the style strength feature to obtain a fusion feature, and generating a stylized image corresponding to the target image according to the fusion feature. The present invention realizes the diversity of stylized image generation, so that the target object in the target image is more coordinated with the style of the stylized image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a stylized image generation method, device, equipment and storage medium. Background Art

[0002] Stylization is a technique that transforms the source style of a real-life image into a target style (e.g., Disney style, anime style, Chinese comic style, etc.) through algorithmic processing. Existing solutions for stylization implementation include directly using a trained network model to generate stylized images. In actual use, no matter what image is input, the style of the stylized image output by the network model remains fixed. At the same time, the style of the stylized image may not match the target object (e.g., a person), resulting in an inconsistency between the image and style of the target object. Summary of the invention

[0003] The embodiments of the present invention provide a stylized image generation method, apparatus, device and storage medium, aiming to solve the technical problem that the style used in the existing stylized image processing technology is fixed, resulting in the inconsistency between the image and style of the target object in the stylized image.

[0004] An embodiment of the present invention provides a method for generating a stylized image, the method comprising:

[0005] Determine a target style type and an attribute feature vector corresponding to a target image to be processed, and determine a style intensity feature corresponding to the target image according to the attribute feature vector;

[0006] Extracting global attribute features of the target image;

[0007] Searching a preset style feature library according to the target style type to obtain style features matching the target style type;

[0008] The global attribute feature, the style feature and the style strength feature are concatenated to obtain a fusion feature, and a stylized image corresponding to the target image is generated according to the fusion feature.

[0009] In one embodiment, a target style type and an attribute feature vector corresponding to a target image to be processed are determined, and a style intensity feature corresponding to the target image is determined according to the attribute feature vector;

[0010] Extracting global attribute features of the target image;

[0011] Searching a preset style feature library according to the target style type to obtain style features matching the target style type;

[0012] The global attribute feature, the style feature and the style strength feature are concatenated to obtain a fusion feature, and a stylized image corresponding to the target image is generated according to the fusion feature.

[0013] In one embodiment, the step of determining a target style type and an attribute feature vector corresponding to a target image to be processed, and determining a style intensity feature corresponding to the target image according to the attribute feature vector comprises:

[0014] Get the pre-trained attribute decoupled encoder;

[0015] The attribute decoupled encoder is used to identify the target image, and a target style type and style strength feature corresponding to the target image are obtained.

[0016] In one embodiment, the training of the attribute decoupling encoder includes:

[0017] Acquire sample object data with attribute annotations, wherein the attribute annotations include attribute labels and style labels of the sample objects;

[0018] Iteratively training an original attribute decoupling encoder using the sample object data, wherein the original attribute decoupling encoder includes a U-Net network connected in sequence and a plurality of three-layer fully connected networks;

[0019] Obtaining the cross loss of each of the fully connected networks after each training, and determining a weighted sum of the cross losses of each of the fully connected networks;

[0020] When the weighted sum is less than or equal to a preset cross loss, the training of the original attribute decoupling encoder is stopped, and the original attribute decoupling encoder whose training is stopped is stored as the attribute decoupling encoder.

[0021] In one embodiment, the step of extracting the global attribute features of the target image includes:

[0022] Get the pre-trained global information encoder;

[0023] The global information encoder is used to identify the target image to obtain global attribute features of the target image.

[0024] In one embodiment, the step of concatenating the global attribute feature, the style feature, and the style strength feature to obtain a fusion feature, and generating a stylized image corresponding to the target image according to the fusion feature includes:

[0025] Based on the channel dimensions of the global attribute feature, the style feature and the style strength feature, superimposing the global attribute feature, the style feature and the style strength feature to obtain the fusion feature;

[0026] Get the pre-trained style decoder;

[0027] The style decoder is used to identify the fusion feature, and a stylized image corresponding to the target image is output.

[0028] In one embodiment, the training of the global information encoder and the style decoder includes:

[0029] Get training style data images with style labels, training style intensity data, and pre-trained attribute decoupling encoders;

[0030] Using the training style data picture to train an original global information encoder, so that the original global information encoder outputs training global attribute features of the training style data picture;

[0031] Acquire a training style feature that matches the style label from a preset style feature library;

[0032] Inputting the training style data picture into the attribute decoupling encoder to obtain a training style intensity feature corresponding to the training style data picture;

[0033] Concatenating the training global attribute feature, the training style feature, and the training style intensity feature to obtain a fused training feature;

[0034] Using the fused training features to train an original style decoder, so that the original style decoder outputs a training stylized picture corresponding to the training style data picture;

[0035] According to the style label, a preset style picture matching the training stylized picture is obtained, and the training stylized picture and the preset style picture are input into a loss function discriminant network to obtain a loss value;

[0036] When the loss value is less than or equal to a preset value, the training of the original global information encoder and the original style decoder is stopped, and the original global information encoder and the original style decoder whose training is stopped are stored as the global information encoder and the style decoder.

[0037] In one embodiment, the stylized image generation method further includes:

[0038] Get the target image to be processed;

[0039] When the target image does not contain a preset trigger action for extracting the target style type, the steps of determining the target style type and the attribute feature vector corresponding to the target image to be processed, and determining the style intensity feature corresponding to the target image according to the attribute feature vector are performed.

[0040] In addition, to achieve the above-mentioned purpose, the present invention further provides a stylized image generation device, the stylized image generation device comprising:

[0041] A first acquisition module is used to determine a target style type and an attribute feature vector corresponding to a target image to be processed, and determine a style intensity feature corresponding to the target image according to the attribute feature vector;

[0042] A second acquisition module is used to extract global attribute features of the target image;

[0043] A third acquisition module is used to search a preset style feature library according to the target style type to obtain a style feature matching the target style type;

[0044] The image generation module is used to splice the global attribute feature, the style feature and the style strength feature to obtain a fusion feature, and generate a stylized image corresponding to the target image according to the fusion feature.

[0045] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal device, which includes: a memory, a processor, and a stylized image generation program stored in the memory and executable on the processor, wherein the stylized image generation program implements the steps of the above-mentioned stylized image generation method when executed by the processor.

[0046] In addition, to achieve the above-mentioned purpose, the present invention also provides a storage medium on which a stylized image generation program is stored, and when the stylized image generation program is executed by a processor, the steps of the above-mentioned stylized image generation method are implemented.

[0047] The technical solutions of a stylized image generation method, device, equipment and storage medium provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0048] The invention adopts the method of determining the target style type and attribute feature vector corresponding to the target image to be processed, determining the style intensity feature corresponding to the target image according to the attribute feature vector, extracting the global attribute feature of the target image, searching the preset style feature library according to the target style type, obtaining the style feature matching the target style type, splicing the global attribute feature, the style feature and the style intensity feature to obtain the fusion feature, and generating the stylized image corresponding to the target image according to the fusion feature, thereby solving the technical problem that the style used in the existing stylized image processing technology is fixed, resulting in the incoordination between the image and style of the target object in the stylized image. The invention generates the stylized image of the target object in the target image according to the multi-dimensional image feature drive, realizes the diversity of the stylized image generation, improves the harmony of the style, and makes the style of the target object and the stylized image more coordinated and more appropriate. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention;

[0050] Figure 2 A schematic diagram of a flow chart of an embodiment of a method for generating a stylized image according to the present invention;

[0051] Figure 3 It is a schematic diagram of the splicing of the fusion features of the present invention;

[0052] Figure 4 Schematic diagram of the specific process of step S210 in the method for generating a stylized image of the present invention;

[0053] Figure 5 A schematic diagram of the network structure of the attribute decoupling decoder of the present invention;

[0054] Figure 6 This is a schematic diagram of a specific process of step S220 of the method for generating a stylized image according to the present invention;

[0055] Figure 7 This is a schematic diagram of a specific process of step S240 of the method for generating a stylized image according to the present invention;

[0056] Figure 8 is a schematic diagram of a global information encoder and a style decoder of the present invention;

[0057] Fig. 9 This is a functional module diagram of the stylized image generation device of the present invention. DETAILED DESCRIPTION

[0058] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0059] like Figure 1 As shown, Figure 1 It is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention.

[0060] It should be noted that Figure 1 This can be a structural diagram of the hardware operating environment of the terminal device.

[0061] As a way to implement this, Figure 1 As shown, the embodiment of the present invention relates to a terminal device, which includes: a processor 1001, such as a CPU, a memory 1002, and a communication bus 1003. The communication bus 1003 is used to realize the connection and communication between these components.

[0062] The memory 1002 may be a high-speed RAX memory or a stable memory (non-volatile XeXory), such as a disk memory. Figure 1 As shown, the memory 1002 as a storage medium may include a stylized image generation program; and the processor 1001 may be used to call the stylized image generation program stored in the memory 1002, and at least perform the following operations:

[0063] Determine a target style type and an attribute feature vector corresponding to a target image to be processed, and determine a style intensity feature corresponding to the target image according to the attribute feature vector;

[0064] Extracting global attribute features of the target image;

[0065] Searching a preset style feature library according to the target style type to obtain style features matching the target style type;

[0066] The global attribute feature, the style feature and the style strength feature are concatenated to obtain a fusion feature, and a stylized image corresponding to the target image is generated according to the fusion feature.

[0067] The embodiments of the present invention provide embodiments of the method for generating a stylized image. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown here.

[0068] like Figure 2 As shown, in the first embodiment of the present invention, the stylized image generation method of the present invention includes the following steps:

[0069] Step S210: determining a target style type and an attribute feature vector corresponding to a target image to be processed, and determining a style intensity feature corresponding to the target image according to the attribute feature vector.

[0070] In this embodiment, the target image to be processed refers to the image that needs to be stylized. The target image can be understood as an image of things in different scenes. For example, the target image is a picture containing a person (the picture needs to include a human face), a picture containing an animal (such as a puppy picture), a picture containing a plant (such as a pine tree picture), and the like. This embodiment and the following other embodiments are explained with the target image being a person picture, that is, the target object included in the person picture is a person, and the person needs to include a human face. The target style type can be understood as the style of the human face, that is, the target style type of the human face, such as Disney style, Japanese comic style, Chinese comic style, hot-blooded anime style, cute comic style, etc. The attribute feature vector is a human face attribute feature vector, which refers to a vector of human face attribute features, used to characterize the human face attribute features of the person. The human face attribute features are, for example, the face shape, expression, eyebrow shape, hairstyle, skin color, age, etc. of the person.

[0071] After obtaining the facial attribute feature vector, the facial style intensity characteristics of the person in the person picture are determined according to the facial attribute feature vector. The facial attribute feature vector has specific data, such as the width of the eyebrow shape, the size of the age, the degree of black and white skin color, etc. The style intensity characteristics are comprehensively represented by the data size of the facial attribute feature vector.

[0072] Step S220: extracting global attribute features of the target image.

[0073] At the same time or after determining the face target style type and face style intensity feature corresponding to the person picture, extract the global attribute feature of the person picture, the global attribute feature is the attribute feature of all information in the person picture, including the face attribute feature and other information. For example, the global attribute feature includes the attribute features of texture, color, edge, content, face, etc. in the person picture.

[0074] Step S230: searching a preset style feature library according to the target style type to obtain style features matching the target style type.

[0075] The preset style feature library is established in advance, and the preset style feature library stores the preset style types and the style features associated with the target style type. After obtaining the character target style type, the face target style type is used as the search condition to search in the preset style feature library, and the preset style type matching the face target style type can be found. The style features associated with the preset style type are the character style features matching the face target style type. For example, if the face target style type is Disney style, the style features associated with the Disney style in the preset style feature library are the ones finally found.

[0076] Furthermore, before executing step S210, the method further includes:

[0077] Get the target image to be processed;

[0078] When the target image does not contain a preset trigger action for extracting the target style type, step S210 is executed to determine the target style type and attribute feature vector corresponding to the target image to be processed, and to determine the style strength feature corresponding to the target image according to the attribute feature vector.

[0079] In order to improve the efficiency of determining the target style type corresponding to the target image, a preset trigger action for extracting the target style type is pre-set, and the preset trigger action and the style features associated with the preset trigger action are stored in the preset style feature library. That is, after obtaining the target image to be processed, it is detected whether the target image contains the preset trigger action for extracting the target style type. If not, step S210 is executed; if contained, the detected trigger action is matched with the preset trigger action stored in the preset style feature library. After obtaining the preset trigger action, the target style type corresponding to the target image can also be obtained, that is, the target style type corresponding to the target image is obtained from the preset style feature library.

[0080] For example, after obtaining the character picture, a human body posture estimation method, such as but not limited to OpenPose, DeepPose, MSPN and other methods, is used to obtain the human body skeleton joint points; according to the triggering rules of the skeleton key points, it is judged whether the character picture contains a preset trigger action for extracting the target face style type. For example, the character picture includes a heart shape with both hands on the top of the head, and the heart shape with both hands on the top of the head matches the cute style stored in the preset style feature library; for another example, the character picture includes a cross of both hands on the chest, and the cross of both hands on the chest matches the Ultraman style. If the character picture contains a preset trigger action for extracting the target face style type, the matching target face style type can be found from the preset style feature library according to the trigger action. If the character picture does not contain a preset trigger action for extracting the target face style type, step S210 is executed. Among them, the identification of the trigger action is to first normalize the human body skeleton points to the backbone skeleton, and then judge whether the character is doing the trigger action through the positional relationship of the skeleton points.

[0081] Step S240: concatenating the global attribute feature, the style feature and the style strength feature to obtain a fusion feature, and generating a stylized image corresponding to the target image according to the fusion feature.

[0082] After obtaining the global attribute features, facial style features and facial style strength features, the global attribute features, facial style features and facial style strength features are spliced ​​to obtain fusion features. The splicing process can be understood as directly adding the global attribute features, facial style features and facial style strength features. Specifically, Figure 3 After obtaining the fusion features, the fusion features include global attribute features, face style features and face style intensity features. According to the fusion features, the preset style picture matching the face style features can be found, and then the fusion features are used to adjust the attribute information in the preset style picture to generate a stylized picture corresponding to the character picture, that is, the generated stylized picture is presented to the user.

[0083] According to the above technical solution, this embodiment generates a stylized image of a target object in a target image based on multi-dimensional image features, thereby achieving diversity in the generation of stylized images, improving the harmony of styles, and making the styles of the target object and the stylized image more coordinated and more appropriate.

[0084] like Figure 4 As shown, based on the above embodiment, step S210 includes the following steps:

[0085] Step S211: obtaining a pre-trained attribute decoupling encoder;

[0086] Step S212: using the attribute decoupling encoder to identify the target image, and obtaining a target style type and style strength features corresponding to the target image.

[0087] The attribute decoupling encoder is pre-trained, including a U-Net network and several three-layer fully connected networks connected in sequence, and the U-Net network is connected to the first layer of the fully connected network in the several fully connected networks, as shown in FIG. Figure 5 As shown. The U-Net network and several three-layer fully connected networks are used to process the character image, so that the penultimate fully connected network in the several fully connected networks outputs the attribute feature vector of the target object, the output attribute feature vector is spliced ​​to obtain a vector matrix, and the vector matrix is ​​input into the last fully connected network in the several fully connected networks to obtain the target style type corresponding to the character image and the feature vector output by the last fully connected network. According to the vector matrix and the feature vector, the probability distribution of the attribute information of the target object is determined, and the style strength feature is determined according to the vector matrix and the probability distribution.

[0088] Based on the character pictures, the training process of the attribute decoupling encoder is explained:

[0089] Since the stylized category is strongly correlated with the image of the face, the present invention proposes a face attribute decoupling encoder, which expands its ability to recognize various attributes of the face and guides its style classification to have a strong correlation with the face attributes. The implementation process of the face attribute decoupling encoder can be understood as two stages, but it is actually end-to-end training. The first stage is to decouple the attribute features of the face, and the second stage is to perform style classification and calculate style strength features based on the decoupled attribute features.

[0090] For the first stage of the task network, the process of decoupling face attributes is to perform multi-attribute classification through a multi-task. Each sub-network corresponds to an attribute classification task. Then the features extracted by the sub-network can represent the attribute. Therefore, the multi-task network can achieve the decoupling of attribute features. The specific description is as follows:

[0091] (1) Using a classification network can realize the extraction of features related to attributes. First, the role of a neural network is to extract features of an image. The deeper the network layer, the more abstract the extracted features. Second, the neural network is combined with a certain task, such as classification and detection, and the features extracted by the deep layer of the network are the features required to complete the task. Therefore, based on these two points, if a neural network model is used to classify a certain attribute of a face, then the features extracted by the neural network are related to the attribute. For example, for the task of age classification, the features extracted by the deep layer of the neural network are related to age, and then the features can be used to represent age.

[0092] (2) The use of a multi-task network structure can achieve the decoupling of facial attribute features; the multi-task adopts a U-Net network (backbone network) + several three-layer fully connected networks (sub-networks). Its purpose can be understood as extracting general features through the backbone network + extracting attribute-focused features through the sub-network, so as to achieve multi-task. General features are the features of the face extracted by the backbone network, such as shallow information such as facial contour and color. Various sub-tasks require such information, so the backbone network of multiple tasks can be shared; attribute-focused features are extracted through sub-networks because the features of various attribute focuses are different. This feature is only for completing sub-tasks, so it only represents the facial attributes corresponding to the sub-tasks.

[0093] Based on the above two points of analysis, a multi-task network can be used to decouple the attribute features of the character image. The feature of each face attribute is the feature vector output by the second-to-last layer of each sub-network;

[0094] For stage 2, style classification is performed. Since the style category is strongly related to the facial attributes, after obtaining the decoupled multiple facial attribute features in process 1, these features are spliced ​​and fused and sent to a classification module of a multi-layer fully connected network to obtain the style category. At the same time, the feature vector of the penultimate layer of the fully connected network is taken as the feature vector containing the facial image and style, referred to as the style feature. Since the existing stylization has the problem of being unable to control the stylization intensity, for example, some cute images should have a more cute and sweet effect after stylization, while some serious middle-aged people should have a more realistic effect after stylization, and even look slightly stylized. In response to such problems, the present invention proposes a method for constructing a style intensity feature vector based on a feedback attention mechanism to control the generated style intensity. This method is different from the general attention calculation method. The present invention adopts a feedback method in the attention distribution calculation and fully considers the style type factor when weighting information.

[0095] The general Attention mechanism strengthens effective information, weakens irrelevant information, solves the problem of model information overload, and improves the ability of neural networks to process information. The essence of the Attention mechanism is actually an addressing process. The implementation process is to give a query vector q related to the task, calculate the attention distribution with the Key vector k and attach it to the Value vector v, so as to calculate the AttentionValue feature vector. At this time, the feature vector is obtained in the training state of the model, and the feature vector also needs to participate in the training of the model.

[0096] The method of constructing a feature vector based on the feedback attention mechanism of the present invention, the query is taken from the last feature vector of the network, the Key=Value vector, and the feature vector of the non-last layer, the purpose is to calculate the feature vector related to the result according to the classification result. At this time, the feature vector is obtained in the deployment state of the model. This feature does not need to participate in the model training, but is used as a style strength feature.

[0097] Therefore, the attention method of the present invention is different from other existing attention methods in the following ways:

[0098] (1) Different usage states: The attention of the present invention is used in the deployment state of the model, and the AttentionValue feature vector is used as a style strength feature and does not participate in model training; while the existing attention is used in the training state of the model, and the AttentionValue feature vector participates in model training.

[0099] (2) Different functions: The attention method of the present invention transcends the network layer and is based on the model output results for feedback. Therefore, the obtained AttentionValue feature vector is only strongly correlated with the model output results. The existing attention is implemented in the same network layer, strengthening the effective information related to the loss function and weakening irrelevant information. Therefore, the obtained AttentionValue feature vector is strongly correlated with all the targets represented by the loss function, rather than just one of them.

[0100] (3) Different scopes: The query, key, and value of the attention method of the present invention come from different network layers; while the query, key, and value of the general attention method are subject to the limitations of model training and generally come from the same layer of the network.

[0101] The method for constructing a feature vector based on the feedback attention mechanism of the present invention is calculated as follows: Step 1, information input: x = [x1, x2, ..., xn], xi represents the feature vector of the i-th face attribute;

[0102] Step 2, attention distribution calculation: q is the feature vector output by the last fully connected network in several fully connected networks, and x is the feature vector output by the second to last fully connected network in several fully connected networks, which means using feedback to perform attention supervision mechanism, and using style type to supervise to obtain facial attribute feature vectors that are more closely related to it. Such feature vectors can better drive the strength of the style as style strength features, and are calculated as follows:

[0103] a=softmax(x T q);

[0104] Where a = [a1, a2, ..., an], which is expressed as attention distribution, also known as probability distribution;

[0105] Step 3. Information price weighting:

[0106] s = ax;

[0107] s represents the style strength feature, and a represents the value obtained by the style classifier at the softmax output, which represents the type probability of the style type.

[0108] The training of the attribute decoupling encoder is end-to-end training. The training process is as follows:

[0109] Acquire sample object data with attribute annotations, where the attribute annotations include attribute labels and style labels of the sample objects;

[0110] The sample object data is used to iteratively train the original attribute decoupling encoder, which includes a U-Net network connected in sequence and several three-layer fully connected networks;

[0111] Get the cross loss of each fully connected network after each training, and determine the weighted sum of the cross loss of each fully connected network;

[0112] When the weighted sum is less than or equal to the preset cross loss, the training of the original attribute decoupling encoder is stopped, and the original attribute decoupling encoder whose training is stopped is stored as the attribute decoupling encoder.

[0113] (1) Dataset preparation, i.e., sample object data with attribute annotations. The sample object data is a face dataset.

[0114] 1) Attribute labels of sample objects, such as labels of facial attributes, including: face shape, expression, hairstyle, eyebrow shape, age and other attribute classification labels;

[0115] 2) Style labels of sample objects, such as style labels of faces, including: each face is labeled with a style that best matches the image, including Disney style, Japanese anime style, Chinese anime style, hot-blooded anime style, cute comic style, etc.

[0116] (2) The attribute decoupling decoder is based on the idea of ​​multi-task learning. The model structure is as follows: Figure 5 As shown in the figure, it can be divided into three parts: the backbone network, the attribute decoupled sub-network, and the classification network for style classification. Part A is a module built based on convolutional neural networks, and part B is a module built based on full connectivity.

[0117] 1) First, the image is input into the backbone network, which uses the U-NET model;

[0118] 2) Then, connect several three-layer full connections, and the task of each full connection is to predict an attribute classification. For example, there are 5 types of face attribute labels, so the number of full connections is also 5, and the number of full connections is deduced in the same way.

[0119] 3) Finally, the feature vectors output by the penultimate layer of these fully connected layers are concatenated and sent to a three-layer fully connected layer for style classification;

[0120] (3) The input of the model is a person image that has undergone conventional pre-processing methods, including RGB channel conversion, scaling, rotation, brightness adjustment, adding different types of noise, and pixel normalization.

[0121] (4) Since they are all classification tasks, the loss function uses cross entropy loss. In multi-task training, each task uses a cross entropy loss, that is, one full connection corresponds to a cross entropy loss. The final loss is the weighted sum of all cross entropy losses, where the loss weight value for face attribute classification is 0.1 and that for style classification is 1. If the weighted sum is less than or equal to the preset cross loss, it means that the error of the original attribute decoupling encoder is very small, then the training of the original attribute decoupling encoder is stopped, and the original attribute decoupling encoder that stops training is stored as the attribute decoupling encoder.

[0122] (5) After training is completed, the style type and style features are obtained.

[0123] Specifically, after obtaining the person image to be processed, the person image is used as the input of the attribute decoupling encoder, and the U-Net network and several three-layer fully connected networks process the person image. Among them, the second-to-last fully connected network ( Figure 5 B1) outputs the facial attribute feature vector, and then concatenates the output facial attribute feature vectors to obtain a vector matrix corresponding to the facial attribute feature vector; then the vector matrix is ​​input into the last layer of the fully connected network ( Figure 5 In B1), the last layer of the fully connected network outputs the target style type corresponding to the character image and also outputs a feature vector.

[0124] After obtaining the vector matrix and the vector matrix, based on the above a=softmax(xT·q) and s=ax, the probability distribution of the facial attribute information is calculated, and the facial style intensity characteristics of the characters in the character image are calculated based on the vector matrix and the probability distribution.

[0125] Among them, it is considered that there may be one person in the character picture, that is, one face, or there may be multiple people, that is, multiple faces. That is, it is necessary to identify whether the target object contained in the target picture is multiple or single. It should be understood that if the target object contained in the target picture is single, the style type output by the last layer of the fully connected network is one, that is, when the target object contained in the target picture is single, the style type output by the last layer of the fully connected network is used as the target style type. If the target object contained in the target picture is multiple, the style type output by the last layer of the fully connected network is also multiple, the number of style types is the same as the number of target objects, and each style type corresponds to a type probability. That is, when the target object contained in the target picture is multiple, the type probability of each style type output by the last layer of the fully connected network is obtained, and then the style type corresponding to the maximum type probability is determined as the target style type.

[0126] like Figure 6 As shown, based on the above embodiment, step S220 includes the following steps:

[0127] Step S221: obtaining a pre-trained global information encoder;

[0128] Step S222: using the global information encoder to identify the target image, and obtaining global attribute features of the target image.

[0129] The global information encoder is pre-trained, and the global information encoder includes a convolution layer, a normalization layer, and an activation function layer. A person picture is taken as input and input into the global information encoder. The person picture is downsampled through the convolution layer in the global information encoder, that is, the person picture is compressed, thereby obtaining a feature map of the person picture; after the feature map is processed by the normalization layer and the activation function layer in the global information encoder in turn, the global attribute features of the target picture are obtained, that is, the global attribute features of the target picture are extracted.

[0130] Among them, the global information encoder implements the encoding of the image content, with the purpose of retaining various detailed information from the global perspective of the image, including global information such as texture, color, edge, content, etc. of the image, as well as detailed information of human objects. The global information encoder retains the global information of the image, and the image can be restored using the global information.

[0131] Figure 7 As shown, based on the above embodiment, step S240 includes the following steps:

[0132] Step S241: Based on the channel dimensions of the global attribute feature, the style feature and the style strength feature, the global attribute feature, the style feature and the style strength feature are superimposed to obtain the fusion feature;

[0133] Step S242: Obtain a pre-trained style decoder;

[0134] Step S243: using the style decoder to identify the fusion features, and outputting a stylized image corresponding to the target image.

[0135] Specifically, the channel dimensions based on the global attribute features, style features, and style strength features can be determined by the matrix dimensions of the vector matrices corresponding to the global attribute features, style features, and style strength features. For example, the matrix dimensions of the vector matrices corresponding to the global attribute features, style features, and style strength features are all 8×8, i.e., 8 channel dimensions. Then, the global attribute features, style features, and style strength features are superimposed one by one according to the channel dimensions to obtain the fusion features.

[0136] After obtaining the fusion feature, the fusion feature is input into the style decoder, and the style decoder identifies the fusion feature to restore the global attribute feature in the fusion feature to obtain the original global attribute information, and at the same time finds the preset style picture that matches the facial style feature according to the facial style feature in the fusion feature. Then, the attribute information of the preset style picture is adjusted according to the original global attribute information and the facial style intensity feature in the fusion feature, so that the stylized person in the adjusted preset style picture is more coordinated, and the facial attribute information of the stylized person is more consistent with the facial attribute information of the original person, thereby improving the harmony of the stylized person.

[0137] In addition to the global information encoder, a style decoder is also pre-trained. The style decoder integrates global information, style features, and style strength features for decoding. The global information can be used to restore the content of the image, and the style information can be used to drive the generation of the target style.

[0138] like Figure 8 As shown in Figure 2, the training process of the global information encoder and style decoder is as follows:

[0139] Get training style data images with style labels, training style intensity data, and pre-trained attribute decoupling encoders;

[0140] Using the training style data picture to train the original global information encoder, so that the original global information encoder outputs the training global attribute features of the training style data picture;

[0141] Obtain training style features matching the style labels from a preset style feature library;

[0142] Input the training style data picture into the attribute decoupling encoder to obtain the training style intensity feature corresponding to the training style data picture;

[0143] The training global attribute features, the training style features and the training style intensity features are concatenated to obtain the fused training features;

[0144] The fused training features are used to train the original style decoder, so that the original style decoder outputs a training stylized picture corresponding to the training style data picture;

[0145] According to the style label, a preset style image matching the training stylized image is obtained, and the training stylized image and the preset style image are input into the loss function discriminant network to obtain a loss value;

[0146] When the loss value is less than or equal to a preset value, the training of the original global information encoder and the original style decoder is stopped, and the original global information encoder and the original style decoder whose training is stopped are stored as the global information encoder and the style decoder.

[0147] (1) Training set preparation: training style data images, i.e., source style data, training style intensity data, i.e., target style data, and preset style feature library.

[0148] 1) Source style data: real-life scene image data containing portraits. The image can contain multiple people, but must contain a human face.

[0149] 2) Target style data: includes data of images in Disney style, Japanese comic style, Chinese comic style, hot-blooded anime style, cute comic style, etc. Each style has different style strengths and the number is evenly distributed. A certain amount of data is taken for each style, for example, 50,000 images.

[0150] (2) Figure 8 As shown in FIG. 1 , it is a system flow chart of a global information encoder and a style decoder. The global information encoder realizes the extraction of global information of an image; the style decoder realizes the generation of a stylized image by integrating global information, style features, and style intensity features.

[0151] (3) The global information encoder extracts a feature map from the image by downsampling. This feature map is equivalent to a compression of the original image information. Its main components are the convolution layer, the normalization layer, and the activation function.

[0152] (4) Style decoder: A structure that upsamples the feature map to the original image. Its main components are the transposed convolution layer (or upsampling layer), the normalization layer and the activation function.

[0153] (5) Network input: three-channel RGB image with a length and width of 512*512, which has undergone conventional pre-processing, including image RGB channel conversion, scaling, rotation, brightness adjustment, adding different types of noise and pixel normalization.

[0154] (6) Network output: three-channel RGB image with a length and width of 512*512.

[0155] (7) Style feature library: It provides style labels and feature vectors containing the style. During the training process, the feature vector is fused with the feature vector of the global information encoder in a channel superposition manner and then sent to the style decoder.

[0156] (8) Training process: A batch of images, for example 12 images, is randomly selected from the source style data as input. Each image passes through the global information encoder to obtain global information features. The image passes through the style decoder to calculate only the style strength features. The image takes out style features from the style feature library according to the style type of the label. The three features are combined and passed through the style decoder to output a batch of images. For each output image, images of the same style are randomly selected from the target data according to its input style type, and finally sent to the discriminant network to calculate the loss.

[0157] (9) Loss function:

[0158] A. Adversarial loss: A conventional discriminant network is used, which consists of 6 convolutional layers and 3 fully connected layers. The loss is as follows:

[0159]

[0160] Where G is the global information encoder and style decoder; D is the discriminator; x->y represents the transition from the source domain to the target style domain;

[0161] B. Cycle consistency loss: Cycle-GAN and Disco-GAN first proposed cycle consistency loss, which is to force the generators to generate images of the other domain in reverse order.

[0162]

[0163] Among them, x->y represents the transition from source domain to target style domain, and y->x represents the transition from target style domain to source domain; |·| 1 represents L1 loss.

[0164] C. Reconstruction loss: Different from the loss function based on domain similarity, the reconstruction loss helps the images generated by the two domain generators to remain consistent in the latent vector space.

[0165]

[0166] All losses are the weighted sum of the three losses:

[0167]

[0168] Among them, λ 1 ,λ2 ,λ 3 are all preset values, such as λ 1 ,λ 2 ,λ 3 =1,10,10. If the loss value is less than or equal to the preset value, it means that the error between the original global information encoder and the original style decoder is already very small, then the training of the original global information encoder and the original style decoder is stopped, and the original global information encoder and the original style decoder whose training is stopped are stored as the global information encoder and the style decoder.

[0169] like Fig. 9 As shown, the present invention provides a stylized image generation device, the stylized image generation device comprising:

[0170] A first acquisition module 310 is used to determine a target style type and an attribute feature vector corresponding to a target image to be processed, and determine a style intensity feature corresponding to the target image according to the attribute feature vector;

[0171] A second acquisition module 320, used to extract global attribute features of the target image;

[0172] A third acquisition module 330 is used to search a preset style feature library according to the target style type to obtain a style feature matching the target style type;

[0173] The image generation module 340 is used to concatenate the global attribute feature, the style feature and the style strength feature to obtain a fusion feature, and generate a stylized image corresponding to the target image according to the fusion feature.

[0174] It should be noted that the stylized image generation device may also include other optional functional modules so that it can perform other steps involved in the above embodiments. Also, the specific implementation of the stylized image generation device of the present invention is basically the same as the above stylized image generation method embodiments, and will not be repeated here.

[0175] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal device, which includes: a memory, a processor, and a stylized image generation program stored in the memory and executable on the processor, wherein the stylized image generation program implements the steps of the above-mentioned stylized image generation method when executed by the processor.

[0176] In addition, to achieve the above-mentioned purpose, the present invention also provides a storage medium on which a stylized image generation program is stored, and when the stylized image generation program is executed by a processor, the steps of the above-mentioned stylized image generation method are implemented.

[0177] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for generating a stylized image, characterized in that: The stylized image generation method comprises: Obtain a pre-trained attribute decoupling encoder; use the attribute decoupling encoder to identify a target image to be processed, and obtain a target style type and style intensity feature corresponding to the target image; Acquire a pre-trained global information encoder; use the global information encoder to identify the target image to obtain global attribute features of the target image; Searching a preset style feature library according to the target style type to obtain style features matching the target style type; Based on the channel dimensions of the global attribute features, the style features and the style strength features, the global attribute features, the style features and the style strength features are superimposed to obtain fused features; a pre-trained style decoder is obtained; the style decoder is used to identify the fused features, and a stylized image corresponding to the target image is output.

2. The method according to claim 1, characterized in that The training of the attribute decoupling encoder includes: Acquire sample object data with attribute annotations, wherein the attribute annotations include attribute labels and style labels of the sample objects; Iteratively training an original attribute decoupling encoder using the sample object data, wherein the original attribute decoupling encoder includes a U-Net network connected in sequence and a plurality of three-layer fully connected networks; Obtaining the cross loss of each of the fully connected networks after each training, and determining a weighted sum of the cross losses of each of the fully connected networks; When the weighted sum is less than or equal to a preset cross loss, the training of the original attribute decoupling encoder is stopped, and the original attribute decoupling encoder whose training is stopped is stored as the attribute decoupling encoder.

3. The method according to claim 1, characterized in that The training of the global information encoder and the style decoder includes: Get training style data images with style labels, training style intensity data, and pre-trained attribute decoupling encoders; Using the training style data picture to train an original global information encoder, so that the original global information encoder outputs training global attribute features of the training style data picture; Acquire a training style feature that matches the style label from a preset style feature library; Inputting the training style data picture into the attribute decoupling encoder to obtain a training style intensity feature corresponding to the training style data picture; Concatenating the training global attribute feature, the training style feature, and the training style intensity feature to obtain a fused training feature; Using the fused training features to train an original style decoder, so that the original style decoder outputs a training stylized picture corresponding to the training style data picture; According to the style label, a preset style picture matching the training stylized picture is obtained, and the training stylized picture and the preset style picture are input into a loss function discriminant network to obtain a loss value; When the loss value is less than or equal to a preset value, the training of the original global information encoder and the original style decoder is stopped, and the original global information encoder and the original style decoder whose training is stopped are stored as the global information encoder and the style decoder.

4. The method according to claim 1, characterized in that The stylized image generation method further includes: Get the target image to be processed; When the target image does not contain a preset trigger action for extracting the target style type, the step of using the attribute decoupling encoder to identify the target image to be processed to obtain the target style type and style strength features corresponding to the target image is performed.

5. A stylized image generation device, characterized in that: The stylized image generating device comprises: A first acquisition module is used to acquire a pre-trained attribute decoupling encoder; use the attribute decoupling encoder to identify a target image to be processed, and obtain a target style type and style intensity feature corresponding to the target image; A second acquisition module is used to acquire a pre-trained global information encoder; use the global information encoder to identify the target image to obtain the global attribute features of the target image; A third acquisition module is used to search a preset style feature library according to the target style type to obtain a style feature matching the target style type; The image generation module is used to superimpose the global attribute features, the style features and the style strength features to obtain fused features based on the channel dimensions of the global attribute features, the style features and the style strength features; obtain a pre-trained style decoder; use the style decoder to identify the fused features, and output a stylized image corresponding to the target image.

6. A terminal device, characterized in that: The terminal device includes: a memory, a processor, and a stylized image generation program stored in the memory and executable on the processor, wherein the stylized image generation program implements the steps of the stylized image generation method according to any one of claims 1 to 4 when executed by the processor.

7. A storage medium, characterized in that: A stylized image generation program is stored thereon, and when the stylized image generation program is executed by a processor, the steps of the stylized image generation method according to any one of claims 1 to 4 are implemented.

Images