Style model generation method, device and computer equipment

Abstract

The present application relates to a style model generation method, apparatus, computer equipment, computer-readable storage medium and computer program product. Embodiments of the present invention can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The method comprises: obtaining the initial style features of the target sample; stylizing the reference sample through a first style model to obtain a first style feature, and stylizing the reference sample through a second style model to obtain a second style feature; the initial style feature and the first style feature correspond to style features of the same style domain; the number of samples trained to obtain the second style model is greater than the number of samples trained to obtain the first style model; based on the initial style feature, the first style feature and the second style feature, the first style model is updated to obtain a target style model corresponding to the target sample. The above method is conducive to expanding the application scenarios of the style model generation method.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a style model generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of computer technology, computer-based image, audio and video processing technologies have also achieved unprecedented development and are widely used in short videos, live broadcasts and other scenarios. Users can select the corresponding style model according to their personal preferences to achieve style conversion of real images, audio or video to improve entertainment effects and highlight personal characteristics.

[0003] The traditional style model generation method performs machine learning on target samples in different style domains and trains the target style model of the corresponding style domain. Since the effect of machine learning is directly related to the number of samples, when the number of samples is small, the target style model generated by the traditional style model generation method cannot accurately describe the characteristics of the corresponding style domain, and there are problems such as model inaccuracy and poor quality of style conversion generation. Therefore, the traditional style model generation method has the disadvantage of limited application scenarios. Summary of the invention

[0004] Based on this, it is necessary to provide a style model generation method, apparatus, computer device, computer-readable storage medium and computer program product that can expand application scenarios in response to the above technical problems.

[0005] A style model generation method, the method comprising:

[0006] Obtain the initial style features of the target sample;

[0007] The reference sample is stylized by a first style model to obtain a first style feature corresponding to the first style model, and the reference sample is stylized by a second style model to obtain a second style feature corresponding to the second style model; the initial style feature and the first style feature correspond to style features of the same style domain; the number of samples obtained by training the second style model is greater than the number of samples obtained by training the first style model, and the second style model has the same model structure as the first style model;

[0008] Based on the initial style feature, the first style feature and the second style feature, the first style model is updated to obtain a target style model corresponding to the target sample.

[0009] In one embodiment, processing each of the sub-style features to obtain the initial style features of the target sample includes:

[0010] The sub-style features are averaged to obtain the initial style features of the target sample.

[0011] In one embodiment, stylizing the reference sample using a second style model to obtain a second style feature corresponding to the second style model includes:

[0012] Performing stylization processing on the reference sample by using a second style model to obtain a second stylized result;

[0013] Extracting the style feature in the second stylization result to obtain the second style feature.

[0014] In one embodiment, the first similarity and the second similarity may both be KL divergence.

[0015] In one embodiment, the interpolation coefficient is a ratio of the first KL divergence to the second KL divergence.

[0016] In one embodiment, the interpolating the model parameters of the second style model using the interpolation coefficient to obtain the target style model corresponding to the target sample includes:

[0017] The interpolation coefficients are used to perform linear interpolation on the model parameters of the second style model to obtain a target style model corresponding to the target sample.

[0018] A style model generation device, the device comprising:

[0019] An initial style feature acquisition module is used to obtain the initial style features of the target sample;

[0020] A model style feature acquisition module, used to perform stylistic processing on a reference sample through a first style model to obtain a first style feature corresponding to the first style model, and to perform stylistic processing on the reference sample through a second style model to obtain a second style feature corresponding to the second style model; the initial style feature and the first style feature correspond to style features of the same style domain; the number of samples obtained by training the second style model is greater than the number of samples obtained by training the first style model, and the second style model has the same model structure as the first style model;

[0021] The target style model determination module is used to update the first style model based on the initial style feature, the first style feature and the second style feature to obtain a target style model corresponding to the target sample.

[0022] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0023] Obtain the initial style features of the target sample;

[0024] The reference sample is stylized by a first style model to obtain a first style feature corresponding to the first style model, and the reference sample is stylized by a second style model to obtain a second style feature corresponding to the second style model; the initial style feature and the first style feature correspond to style features of the same style domain; the number of samples obtained by training the second style model is greater than the number of samples obtained by training the first style model, and the second style model has the same model structure as the first style model;

[0025] Based on the initial style feature, the first style feature and the second style feature, the first style model is updated to obtain a target style model corresponding to the target sample.

[0026] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0027] Obtain the initial style features of the target sample;

[0028] The reference sample is stylized by a first style model to obtain a first style feature corresponding to the first style model, and the reference sample is stylized by a second style model to obtain a second style feature corresponding to the second style model; the initial style feature and the first style feature correspond to style features of the same style domain; the number of samples obtained by training the second style model is greater than the number of samples obtained by training the first style model, and the second style model has the same model structure as the first style model;

[0029] Based on the initial style feature, the first style feature and the second style feature, the first style model is updated to obtain a target style model corresponding to the target sample.

[0030] A computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0031] Obtain the initial style features of the target sample;

[0032] The reference sample is stylized by a first style model to obtain a first style feature corresponding to the first style model, and the reference sample is stylized by a second style model to obtain a second style feature corresponding to the second style model; the initial style feature and the first style feature correspond to style features of the same style domain; the number of samples obtained by training the second style model is greater than the number of samples obtained by training the first style model, and the second style model has the same model structure as the first style model;

[0033] Based on the initial style feature, the first style feature and the second style feature, the first style model is updated to obtain a target style model corresponding to the target sample.

[0034] In the above-mentioned style model generation method, apparatus, computer device, computer-readable storage medium and computer program product, in the process of determining the target style model of the target sample, the first style model corresponding to the same style domain as the target sample is updated based on the second style model with the same model structure. Since the number of samples obtained by training the second style model is greater than the number of samples obtained by training the first style model, the second style model has higher accuracy than the first style model. Based on this, the target style model is obtained by updating the first style model using the above-mentioned method, which can improve the accuracy of the target style model, and to a certain extent, the generation quality of the target style model is no longer restricted by the number of samples, which is conducive to expanding the application scenarios of the style model generation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a flow chart of a style model generation method in one embodiment;

[0036] Figure 2 A schematic diagram of a process of updating a first style model based on an initial style feature, a first style feature, and a second style feature to obtain a target style model corresponding to a target sample in an embodiment;

[0037] Figure 3 A schematic diagram of a process for obtaining initial style features of a target sample in one embodiment;

[0038] Figure 4 A schematic diagram of a process of extracting style features corresponding to feature information through a preset style extraction network to obtain initial style features of a target sample in one embodiment;

[0039] Figure 5 A schematic diagram of a process of performing stylization processing on a reference sample by using a first style model to obtain a first style feature corresponding to the first style model in an embodiment;

[0040] Figure 6 A schematic diagram of a process of performing stylization processing on a reference sample by using a second style model to obtain a second style feature corresponding to the second style model in an embodiment;

[0041] Figure 7 is a schematic diagram of a reference sample and its corresponding first stylization result and second stylization result in one embodiment;

[0042] Figure 8 A schematic diagram of a process for training a generative adversarial network model in one embodiment;

[0043] Fig. 9 A schematic diagram of a flow chart of determining a learning rate corresponding to a training sample according to the number of training samples used to train a generative adversarial network model in one embodiment;

[0044] Fig.10 is a schematic diagram of a training architecture of a first style model in one embodiment;

[0045] Fig.11 is a schematic diagram of an updating process of a first style model in an embodiment;

[0046] Fig.12 It is a structural block diagram of a style model generating device in one embodiment;

[0047] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] A style model refers to a model that corresponds to a specific style domain, grasps the style characteristics of the style domain, and the rules for style conversion of samples in the style domain. It can also stylize the samples based on the rules to obtain a stylized result corresponding to the samples with the style characteristics of the style domain.

[0050] The traditional style model generation method uses training samples of different style domains to perform model training to obtain style models corresponding to different style domains. The number of samples of each training sample is a predetermined data amount to ensure the generation quality of each corresponding style model. The user inputs the original data such as image, audio or video into the style model of a specific style domain to achieve the stylized conversion result of the original data in the style domain. However, in actual application, there are always some style domains where it is difficult to obtain training samples of the predetermined data amount, or the cost of generating training samples is high, resulting in a small number of samples in the style domain. If the parameters and algorithms that are exactly the same as the training samples of the predetermined data amount are directly used to train the model of the target samples with a small number of samples, there may be problems of difficulty in training or overfitting; if the target style model is obtained by only adjusting the parameters of the model training to match the target samples with a small number of samples, it will be limited by the number of samples of the target samples, resulting in the target style model being unable to accurately describe the characteristics of the corresponding style domain.

[0051] Based on this, the present application proposes a style model generation method. Taking into account that the number of samples trained to obtain the second style model is greater than the number of samples trained to obtain the first style model, and the model structures of the first style model and the second style model are the same, the first style model corresponding to the target sample is updated based on the second style model to obtain a target style model. This can improve the quality of the target style model corresponding to the target sample with a smaller number of samples and expand the application scenarios of the style model generation method.

[0052] It can be understood that the method involved in the present application can be applied to the generation of style models of different types of data samples such as images, audio or video. For example, images can include photos, paintings, and film and television pictures. Image style models can be different animation styles, painting styles, or photography styles. Taking painting style as an example, a variety of painting styles can be divided based on painting materials, such as Chinese painting, gouache painting, sketching, and oil painting; painting styles can also be divided based on painting techniques, such as fine brushwork style, freehand style, abstract style, and realistic style; and a variety of painting styles can be divided based on the artist's unique painting style, such as "Van Gogh style", "Xu Beihong style" and "Picasso style". Similarly, the audio style can include the timbre style of the sound in the audio and the music style corresponding to the background music. For example, the timbre style can be the voice style of children's voices, male voices, female voices, or a specific cartoon character; the music style can be pop music style, jazz style, or rock music style, or it can be piano style, electronic keyboard style, erhu style, and so on. Furthermore, video data samples generally include image samples and audio samples, and therefore, the video style model can be formed by combining the above-mentioned image style model and audio style model, which will not be described in detail here.

[0053] In one embodiment, Figure 1 As shown, a style model generation method is provided. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc. The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. In this embodiment, the method includes the following steps:

[0054] Step S101, obtaining the initial style features of the target sample.

[0055] The target sample refers to the sample for which the target style model needs to be generated. As mentioned above, the specific type of the target sample is not unique, for example, it can be an image sample, an audio sample or a video sample. The style feature refers to the information that can reflect the style characteristics of the target sample. Taking the image sample as an example, the style characteristics of the target sample can include color, texture, shape and spatial relationship, etc. The initial style feature of the target sample refers to the style feature directly extracted from the target sample without performing style conversion processing on the target sample. Furthermore, the initial style feature can be in the form of a vector or a matrix.

[0056] Specifically, the terminal can extract the style features of the target sample to obtain the initial style features of the target sample. For example, various machine learning network models can be used to extract the style features of the target sample to obtain the initial style features of the target sample. For another example, when the target sample is an image sample, and the style features of the image sample are mainly color features, the terminal can extract the color channel value of the target sample on each color channel, and then perform convolution and weighted sum operations on each color channel value to obtain the initial style features of the target sample.

[0057] Step S102, stylizing the reference sample using the first style model to obtain a first style feature corresponding to the first style model, and stylizing the reference sample using the second style model to obtain a second style feature corresponding to the second style model.

[0058] Among them, the reference sample is consistent with the target sample in type, and the first style feature and the second style feature are consistent with the form of the initial style feature. That is, when the target sample is an image sample, the reference sample is also an image sample; when the initial style feature is a vector, the first style feature and the second style feature are also vectors. It should be noted that in this embodiment, it is only necessary to ensure that the type of the reference sample is consistent with that of the target sample, and there is no requirement for the style domain corresponding to the reference sample. Taking the image sample as an example, the reference sample can specifically be a real photo, oil painting, cartoon, gouache painting, etc. Furthermore, the first style feature refers to information that can reflect the style characteristics of the first style model; the second style feature refers to information that can reflect the style characteristics of the second style model.

[0059] Specifically, by stylizing the reference sample with the first style model, the first style feature corresponding to the first style model can be obtained. Similarly, by stylizing the reference sample with the second style model, the second style feature corresponding to the second style model can be obtained. The number of samples trained to obtain the second style model is greater than the number of samples trained to obtain the first style model, and the model structure of the second style model is the same as that of the first style model. It should be noted that the initial style feature and the first style feature correspond to style features of the same style domain, that is, the first style feature and the initial style feature are both style features of the first style domain corresponding to the target sample. The difference lies in the different ways of obtaining the two: the initial style feature is directly extracted from the target sample, while the first style feature is obtained by combining the reference sample and the first style model. Therefore, the initial style feature must be able to reflect the style characteristics of the target sample, and the degree of fit between the first style feature and the style characteristics of the target sample is related to the accuracy of the first style model.

[0060] Furthermore, in the process of training the model, the number of samples obtained by training the model is directly related to the accuracy of the model. The larger the number of samples, the smaller the error and the higher the model accuracy. Conversely, the smaller the sample size, the larger the error and the lower the model accuracy. Based on this, in one embodiment, the number of samples obtained by training the second style model differs from the number of samples obtained by training the first style model by more than a preset difference. The difference can be specifically represented by the difference between the number of samples obtained by training the second style model and the number of samples obtained by training the first style model, or by the ratio of the number of samples obtained by training the second style model to the number of samples obtained by training the first style model. The preset difference can be determined based on the actual number of samples of different style models. Specifically, for the target sample, when there are multiple second style models, a second style model with a number of samples greater than a preset difference from the first style model can be selected to ensure that the accuracy of the target style model is improved after the model is updated.

[0061] Step S103: updating the first style model based on the initial style feature, the first style feature and the second style feature to obtain a target style model corresponding to the target sample.

[0062] Specifically, the specific manner of updating the first style model based on the initial style feature, the first style feature, and the second style feature is not unique.

[0063] In one embodiment, based on the initial style feature, the first style feature and the second style feature, an incremental sample corresponding to the target sample is obtained, and the first style model is updated based on the incremental sample.

[0064] Specifically, the terminal interpolates the initial style features based on the similarity between the initial style features and the first style features, and the similarity between the initial style features and the second style features, to obtain the incremental sample style features of the first style domain corresponding to the initial style features. Then, based on the incremental sample style features, an incremental sample is obtained, and the incremental sample is used to perform incremental training on the first style model to obtain an updated first style model, which is the target style model corresponding to the target sample.

[0065] In another embodiment, Figure 2 As shown, step S103 includes:

[0066] Step S201 : determining an interpolation coefficient, wherein the interpolation coefficient is determined according to a first similarity between an initial style feature and a first style feature, and a second similarity between the initial style feature and a second style feature.

[0067] The first similarity and the second similarity may be in the form of KL divergence, JS divergence or mean square error. Taking KL divergence as an example, the smaller the KL divergence of two style features, the higher the similarity, and vice versa, the lower the KL divergence, the lower the similarity. The interpolation coefficient may be the ratio of the first similarity to the second similarity, or the ratio of the second similarity to the first similarity.

[0068] Specifically, the terminal may calculate a first similarity between the initial style feature and the first style feature, and a second similarity between the initial style feature and the second style feature, and then determine a difference coefficient according to the first similarity and the second similarity.

[0069] Step S202: interpolating the model parameters of the first style model using the interpolation coefficients to obtain a target style model corresponding to the target sample.

[0070] Specifically, according to the interpolation coefficients, the weights of the model parameters of the first style model and the second style model can be determined, and then based on the corresponding model parameters and weights, the updated model parameters of the first style model can be obtained to obtain the target style model corresponding to the target sample.

[0071] In one embodiment, step S202 includes: performing linear interpolation on model parameters of the second style model using interpolation coefficients to obtain a target style model corresponding to the target sample.

[0072] Among them, linear interpolation refers to an interpolation method in which the interpolation function is a first-order polynomial, and the interpolation error at the interpolation node is zero. Compared with other interpolation methods, such as parabolic interpolation, linear interpolation is simple and convenient. Specifically, the corresponding model parameters and weights can be weighted summed to obtain the model parameters of the updated first style model, and the target style model corresponding to the target sample can be obtained. For example, when the interpolation coefficient is the ratio of the first similarity to the second similarity, the model parameter of the target style model = interpolation coefficient * model parameter of the first style model + (1-interpolation coefficient) * model parameter of the second style model. The first style model is updated by interpolation, and the algorithm is simple, which is conducive to improving the working efficiency of the style model generation method.

[0073] Furthermore, for the target sample, when there are multiple second style models, after calculating the second similarity between the initial style feature and each second style feature, the second style feature with the highest similarity to the initial style feature and its corresponding second style model can be selected to update the first style model to obtain the target style model corresponding to the target sample, so as to improve the accuracy of the target style model. For example, the target sample is an oil painting sample, and the number of training samples of the gouache style model and the comic style model is greater than the number of training samples of the first style model in the oil painting style domain, and both can be used as the second style model. By calculating the similarity between the oil painting style feature and the gouache style feature, as well as the oil painting style feature and the comic style feature, it can be determined that the oil painting style feature has a higher similarity with the gouache style feature, and the gouache style model can be selected as the second style model to update the original oil painting style model to improve the model update effect.

[0074] In the above style model generation method, in the process of determining the target style model of the target sample, the first style model corresponding to the same style domain as the target sample is updated based on the second style model with the same model structure. Since the number of samples obtained by training the second style model is greater than the number of samples obtained by training the first style model, the second style model has higher accuracy than the first style model. Based on this, the target style model is obtained by updating the first style model using the above method, which can improve the accuracy of the target style model. To a certain extent, the generation quality of the target style model is no longer restricted by the number of samples, which is conducive to expanding the application scenarios of the style model generation method.

[0075] In one embodiment, the target sample includes at least two sample data. In the case of this embodiment, Figure 3 As shown, step S101 includes:

[0076] Step S301, extracting feature information of each sample data in the target sample.

[0077] The characteristic information of the sample data refers to information that can reflect the characteristics of the sample data in certain dimensions, for example, brightness characteristic information, texture characteristic information, and color characteristic information of the image sample.

[0078] Specifically, a backbone network for extracting feature information can be constructed based on a machine learning algorithm, and each sample data is input into the backbone network respectively. The backbone network extracts feature information from the input target sample and outputs feature information of each sample data.

[0079] Step S303: extracting style features corresponding to the feature information through a preset style extraction network to obtain initial style features of the target sample.

[0080] The style extraction network is constructed based on a machine learning algorithm. Specifically, the feature information is input into a preset style extraction network, and after the style extraction network processes the feature information, the style features corresponding to the feature information can be obtained, and then the initial style features of the target sample can be obtained. Furthermore, the style extraction network processes the feature information, including but not limited to feature decomposition, linear operation, and normalization processing.

[0081] It should be noted that there is no unique way to obtain the initial style features of the target sample.

[0082] In one embodiment, Figure 4 As shown, step S303 includes:

[0083] Step S401: extract sub-style features corresponding to feature information of each sample data through a preset style extraction network.

[0084] Specifically, the feature information of each sample data is respectively input into a preset style extraction network, and after the style extraction network processes the feature information, the sub-style features corresponding to each sample data can be obtained.

[0085] Step S402: Process each sub-style feature to obtain the initial style feature of the target sample.

[0086] Specifically, there is no unique way to process each sub-style feature to obtain the initial style feature of the target sample. For example, each sub-style feature can be averaged to obtain the initial style feature of the target sample; or correlation analysis can be performed on each sub-style feature, noise features can be removed, and then the remaining sub-style features can be averaged to obtain the initial style feature of the target sample.

[0087] In another embodiment, after step S301 and before step S303, the process further includes: averaging sub-feature information of the same type in the feature information of each sample data to obtain feature information of the target sample.

[0088] As mentioned above, the characteristic information of sample data refers to information that can reflect the characteristics of sample data in certain dimensions. The same type of sub-characteristic information in the characteristic information of each sample data refers to information in the characteristic information of each sample data that reflects the characteristics in the same dimension.

[0089] Specifically, after extracting the feature information of each sample data in the target sample, feature information corresponding to multiple sample data will be obtained. The type of sub-feature information contained in the feature information of each sample data is the same, but the specific values ​​are different. Taking the image sample as an example, feature information 1 of sample data 1 and feature information 2 of sample data 2 both contain different types of sub-feature information such as brightness feature information, texture feature information, and color feature information. Based on this, the feature information of the same type in the feature information of each sample data is averaged to obtain the feature information of the target sample. Correspondingly, step S303 includes: extracting the style features corresponding to the feature information of the target sample through a preset style extraction network to obtain the initial style features of the target sample. That is, in the case of this embodiment, there is no need to extract the sub-initial style features of each sample data. It is only necessary to input the feature information of the target sample obtained after averaging into the style extraction network to obtain the initial style features of the target sample.

[0090] In the above embodiment, when the target sample includes at least two sample data, the initial style feature of the target sample is obtained based on the feature information of each sample data, which is conducive to improving the accuracy of the initial style feature.

[0091] In one embodiment, Figure 5 As shown, stylizing the reference sample by using the first style model to obtain a first style feature corresponding to the first style model includes:

[0092] Step S501: stylize a reference sample using a first style model to obtain a first stylized result.

[0093] Among them, stylization processing is style conversion processing, which is the process of converting the reference sample from the original style to the first style corresponding to the first style model. The reason why the first style model can realize the stylization processing of the sample is that before the first style model is put into use, the developer uses the first training sample of the first style domain corresponding to the target sample to train the first style model, so that the first style model can master the law of converting the sample to the first style, and can perform stylization processing on the sample based on the law to obtain the first stylization result. It can be understood that the first stylization result is the reference sample after the first style conversion. For example, when the reference sample is a real face picture and the first style model is an oil painting style model, the first stylization result is a face picture in the oil painting style.

[0094] Step S502: extracting style features of the first stylization result to obtain first style features.

[0095] Specifically, the terminal may extract style features from the first stylized result to obtain first style features of the first stylized result corresponding to the first style model. For the specific method of extracting style features from the first stylized result, refer to the method of extracting initial style features from the target sample above, which will not be described here.

[0096] Furthermore, as described above, the second style model has the same model structure as the first style model. Based on this, in one embodiment, Figure 6 As shown, stylizing the reference sample by using the second style model to obtain a second style feature corresponding to the second style model includes:

[0097] Step S601: stylize the reference sample using a second style model to obtain a second stylized result.

[0098] Step S602: extracting style features of the second stylization result to obtain second style features.

[0099] For specific definitions of step S601 and step S602, please refer to the definitions of step S501 and step S502 above, which will not be repeated here.

[0100] like Figure 7 FIG. 1 is a schematic diagram of a reference sample and its corresponding first stylized result and second stylized result in one embodiment. The reference sample is a real face picture, the first stylized result is obtained by stylizing the real face picture using a first style model corresponding to a first animation domain, and the second stylized result is obtained by stylizing the real face picture using a second style model corresponding to a second animation domain.

[0101] In the above embodiment, a method for acquiring the first style feature and the second style feature is provided, and the algorithm is simple, which is conducive to improving the working efficiency of the style model generation method.

[0102] It should be noted that the specific model types of the first style model and the second style model are not unique. For example, the first style model and the second style model may be the same type of machine learning network model. The machine learning network model may be a convolutional neural network model, a deconvolutional neural network model, a deep neural network model, or a generative adversarial network model, etc.

[0103] In one embodiment, the first style model and the second style model include generative adversarial network models.

[0104] The generative adversarial network model includes a generator network, a discriminator network, and a style generation network. The style generation network is used to provide style features of the corresponding style domain, the generator network is used to generate content based on the style features, and the discriminator network is used to discriminate the content generated by the generator network. The core logic of the generative adversarial network is the process of confrontation and competition between the generator network and the discriminator network.

[0105] Specifically for the present application, the case where the sample type is an image sample is taken as an example. The training sample is input into the style generation network, and the image style features of the training sample are obtained by the style generation network, that is, the image style features of the style domain corresponding to the training sample. The generation network is used to convert the image style of the original image based on the image style features to obtain a style-converted image. The discriminant network can discriminate the image after style conversion: if the style of the image after style conversion meets the discrimination conditions, it can pass the discriminant network; if the style of the image after style conversion does not meet the discrimination conditions, it cannot pass the discrimination of the discriminant network.

[0106] Furthermore, the generation network can be composed of an encoder and a decoder. The encoder and the decoder can both be machine learning networks. For example, the encoder and the decoder can both be CNN (Convolutional Neural Networks) architectures. The encoder is used to encode the original image to obtain the image information of the original image. The input of the decoder is the output of the encoder and the image style features provided by the style generation network. The decoder uses the common AdaIN algorithm to perform style transfer of the target style, and the output of the decoder is the stylized result of the original image.

[0107] In one embodiment, the first style model and the second style model include a generative adversarial network model. In this embodiment, Figure 8 As shown, the training methods for generating adversarial network models include:

[0108] Step S801, determining a learning rate corresponding to the training sample according to the number of training samples used to train the generative adversarial network model.

[0109] Among them, the learning rate, also known as the learning rate, is a hyperparameter that controls the gradient of the loss function to adjust the network weights during model training. It usually represents the speed of information accumulation in the neural network over time. The loss function refers to the function that calculates the difference between the predicted output calculated by the neural network and the actual output. During the model training process, the learning rate can be constant or gradually change with the number of iterations. For ease of understanding, the following example uses the case where the learning rate is constant as an example.

[0110] Specifically, the learning rate determines whether the objective function can converge to the local minimum and when it converges to the minimum. If the learning rate is too large, the loss function may directly cross the global optimal point, which may easily cause gradient explosion, large vibration amplitude of the loss function, and difficulty in converging the model. If the learning rate is too small, the loss function changes very slowly, which is prone to overfitting, and will greatly increase the convergence complexity of the network. Based on this, the learning rate that matches the specific situation of the training sample can be determined. Specifically, the learning rate corresponding to the training sample can be determined according to the number of training samples used to train the generative adversarial network model.

[0111] As described above, the number of samples of the first training samples used to train the first style model is different from the number of samples of the second training samples used to train the second style model, so the first learning rate corresponding to the first training sample is also different from the second learning rate corresponding to the second training sample. Specifically, the first learning rate corresponding to the first training sample can be determined according to the first sample number of the first training samples used to train the first style model, and the second learning rate corresponding to the second training sample can be determined according to the second sample number of the second training samples used to train the second style model.

[0112] In one embodiment, Fig. 9 As shown, step S901 includes:

[0113] Step S901, according to the sample number of training samples used to train the generative adversarial network model, calculate the ratio of the sample number to the preset data amount.

[0114] Among them, the preset data volume is an empirical value preset by the developer, which can be determined comprehensively based on factors such as sample type, sample capacity required for machine learning, and historical model training conditions. When the sample type is determined, multiple model training experiments are conducted based on training samples of the preset data volume to determine the optimal learning rate of the preset data volume and obtain the preset learning rate corresponding to the preset data volume. Specifically, based on the first sample number of the first training sample used to train the first style model, the ratio of the first sample number to the preset data volume can be calculated; based on the second sample number of the second training sample used to train the second style model, the ratio of the second sample number to the preset data volume can be calculated.

[0115] Step S902: Determine the learning rate corresponding to the training sample according to the ratio and the preset learning rate corresponding to the preset data amount.

[0116] Specifically, the first learning rate corresponding to the first training sample can be determined based on the ratio of the first sample number to the preset data amount, and the preset learning rate corresponding to the preset data amount; the second learning rate corresponding to the second training sample can be determined based on the ratio of the second sample number to the preset data amount, and the preset learning rate corresponding to the preset data amount.

[0117] Furthermore, a mapping relationship table between the ratio range and the reference learning rate can be established, wherein the reference learning rate is the product of the preset learning rate and the reference coefficient. Then, according to the ratio calculated in step S901, the corresponding reference learning rate is determined by looking up the table to obtain the learning rate corresponding to the training sample. Alternatively, the ratio of the number of samples to the preset data volume can be multiplied by the preset learning rate to obtain the learning rate corresponding to the training sample.

[0118] Step S802: Based on the learning rate, the training sample is used to perform model training on the generative adversarial network model to obtain a style model of the style domain corresponding to the training sample.

[0119] Specifically, based on the first learning rate, the generative adversarial network model is trained using the first training sample, so as to obtain a first style model that masters the law of performing a first style conversion on the sample and can perform stylized processing on the sample based on the law; based on the second learning rate, the generative adversarial network model is trained using the second training sample, so as to obtain a second style model that masters the law of performing a second style conversion on the sample and can perform stylized processing on the sample based on the law.

[0120] In the above embodiment, during the training phase, a learning rate that matches the number of training samples is used to perform model training to obtain a first style model and a second style model, which is beneficial to improving the scientific nature of the learning rate determination process, improving the model training effect, and avoiding overfitting.

[0121] For ease of understanding, the following Fig.10 and Fig.11 , the style model generation method is explained in detail.

[0122] In one embodiment, the sample type of the target sample is a face picture. Both the first style model and the second style model are generative adversarial network models, and the first style model and the second style model correspond to different animation style domains. In the case of this embodiment, Fig.10 As shown in Figure 1, the generative adversarial network model includes a generative network, a discriminative network, and a style generation network. The generative network includes an encoder and a decoder, and the structures of the encoder and the decoder are symmetrical.

[0123] Specifically, a real face is used as the input of the encoder, and the encoder extracts the portrait information of the real face, and the portrait information is input into the decoder. The first animation training sample of the first animation style domain is used as the input of the style generation network, and the style generation network extracts the style vector of the first animation training sample. The style generation network specifically includes a backbone network and a style extraction network. The first animation training sample can be input into the backbone network, and the backbone network converts the 3*256*256 input into a 512*8*8 deep feature, and then the deep feature is sent to the corresponding sub-style extraction network (if there are n types of animation styles, then there are also n sub-style extraction networks, and the structure of each sub-style is consistent), and the output is a 64*1*1 one-dimensional style vector, which is the style vector of the corresponding first animation domain style. The decoder combines the portrait information output by the encoder with the style vector provided by the style generation network to obtain a first stylized result, which is the face image information with the first animation style. The first stylized result has the first animation style, as well as the expression, posture and identity information of the real portrait.

[0124] The decoder outputs the first stylization result to the discriminant network, which can discriminate the style-converted image: if the style of the style-converted image meets the discrimination conditions, it can pass the discriminant network; if the style of the style-converted image does not meet the discrimination conditions, it cannot pass the discrimination of the discriminant network. For example, if the received picture is a real animated picture, such as one drawn by hand or generated by PS, a discrimination result of 1 is output; if the received picture is an animated picture generated by neural network reasoning and learning, a discrimination result of 0 is output. These two types of data are used as inputs to the discriminator network, with real animated pictures as positive samples and animated pictures generated by the network as negative samples. The purpose of the discriminant network is to discriminate the difference between the two and force the network to learn the generation method of real animation. For each picture in the anime style domain, there will be a separate discriminant network to discriminate, and it will be trained as a discriminant network corresponding to a certain anime style domain.

[0125] In order to solve the problem that training samples with a small number of samples are difficult to train, the learning rate of the training sample is determined according to the number of samples of the training sample, the preset data volume and the preset learning rate corresponding to the preset data volume. For example, if the number of samples of the first training sample in the first animation style domain is less than the number of samples of the second training sample in the second animation style domain, then the first learning rate corresponding to the first training sample is less than the second learning rate corresponding to the second training sample. For another example, a normal learning rate is used when training a style model based on a sufficient number of training samples, and a lower learning rate is used when training a style model based on a small number of training samples to prevent overfitting learning of the network. Among them, a sufficient number of training samples refers to training samples with a sample number greater than a set value, and a small number of training samples refers to training samples with a sample number less than a preset value. The set value is greater than the preset value.

[0126] Further, based on the corresponding learning rate, the first training sample is used for model training to obtain a first style model B of the first animation domain, and the second training sample is used for model training to obtain a second style model A of the second animation domain. The number of samples of the second training sample is greater than the number of samples of the first training sample.

[0127] like Fig.11 As shown, in the test phase, the target sample of the first animation domain is sent to the style generation network to obtain the initial style vector x of the target sample. The real face picture is stylized by the first style model B to obtain the first style picture of the first animation domain of the real face picture, and then the first style picture is sent to the style generation network to obtain the first style vector b. The same real face picture is stylized by the second style model A to obtain the second style picture of the second animation domain of the real face picture, and then the second style picture is sent to the style generation network to obtain the second style vector a. The first divergence DKL(x||b) of the initial style vector x and the first style vector b, and the second divergence DKL(x||b) of the initial style vector x and the second style vector a are calculated respectively. KL (x||a), and the ratio of the first divergence to the second divergence is determined as the interpolation coefficient. Finally, the model parameters of the first style model b are interpolated based on the interpolation coefficient to obtain the target style model X corresponding to the target sample.

[0128] Since the number of samples obtained by training the second style model A is greater than the number of samples obtained by training the first style model B, the second style model A has higher accuracy than the first style model B corresponding to the target sample. Based on this, the target style model X is obtained by updating the first style model B using the above method, which can improve the accuracy of the target style model X. To a certain extent, the generation quality of the target style model is no longer restricted by the number of samples, which is conducive to expanding the application scenarios of the style model generation method.

[0129] In the above embodiments, on the one hand, in the training phase, a learning rate that matches the number of training samples is used to perform model training to obtain a first style model and a second style model, which is beneficial to improving the scientific nature of the learning rate determination process, improving the model training effect, and avoiding overfitting. On the other hand, in the testing phase, considering that the number of samples used to train the second style model is greater than the number of samples used to train the first style model, and the model structures of the first style model and the second style model are the same, the first style model corresponding to the target sample is updated based on the second style model to obtain a target style model, which can improve the quality of the target style model corresponding to the target sample with a smaller number of samples and expand the application scenarios of the style model generation method.

[0130] It should be understood that, although each step in each flow chart involved in the above-described embodiment is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each flow chart involved in the above-described embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0131] In one embodiment, Fig.12 As shown, a style model generation device 1200 is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: an initial style feature acquisition module 1201, a model style feature acquisition module 1202 and a target style model determination module 1203, wherein:

[0132] An initial style feature acquisition module 1201 is used to acquire the initial style features of the target sample;

[0133] The model style feature acquisition module 1202 is used to perform stylistic processing on the reference sample through the first style model to obtain a first style feature corresponding to the first style model, and to perform stylistic processing on the reference sample through the second style model to obtain a second style feature corresponding to the second style model; the initial style feature and the first style feature correspond to style features of the same style domain; the number of samples obtained by training the second style model is greater than the number of samples obtained by training the first style model, and the model structure of the second style model is the same as that of the first style model;

[0134] The target style model determination module 1203 is used to update the first style model based on the initial style feature, the first style feature and the second style feature to obtain a target style model corresponding to the target sample.

[0135] In one embodiment, the target sample includes at least two sample data. In this embodiment, the initial style feature acquisition module 1201 includes a characteristic information extraction unit and an initial style feature determination unit, wherein:

[0136] A characteristic information extraction unit, used to extract characteristic information of each sample data in the target sample;

[0137] The initial style feature determination unit is used to extract the style features corresponding to the feature information through a preset style extraction network to obtain the initial style features of the target sample.

[0138] In one embodiment, the initial style feature acquisition module 1201 further includes: a feature information processing module, which is used to average the sub-feature information of the same type in the feature information of each sample data to obtain the feature information of the target sample. In the case of this embodiment, the initial style feature determination unit is specifically used to: extract the style feature corresponding to the feature information of the target sample through a preset style extraction network to obtain the initial style feature of the target sample.

[0139] In one embodiment, the initial style feature determination unit includes a sub-style feature extraction component and an initial style feature determination component, wherein:

[0140] A sub-style feature extraction component is used to extract sub-style features from feature information of each sample data through a preset style extraction network;

[0141] The initial style feature determination component is used to process each sub-style feature to obtain the initial style feature of the target sample.

[0142] In one embodiment, the model style feature acquisition module 1202 includes a first stylization result acquisition unit and a first style feature extraction unit, wherein:

[0143] A first stylized result acquisition unit, configured to perform stylized processing on the reference sample using a first style model to obtain a first stylized result;

[0144] The first style feature extraction unit is used to extract the style feature of the first stylization result to obtain the first style feature.

[0145] In one embodiment, the model style feature acquisition module 1202 further includes a second stylization result acquisition unit and a second style feature extraction unit, wherein:

[0146] A second stylized result acquisition unit, configured to perform stylized processing on the reference sample using a second style model to obtain a second stylized result;

[0147] The second style feature extraction unit is used to extract the style feature of the second stylization result to obtain the second style feature.

[0148] In one embodiment, the first style model and the second style model include a generative adversarial network model. In this embodiment, the style model generation device 1200 further includes a learning rate determination module and a style model training module, wherein:

[0149] A learning rate determination module, used to determine a learning rate corresponding to a training sample according to the number of training samples used to train a generative adversarial network model;

[0150] The style model training module is used to perform model training on the generative adversarial network model based on the learning rate and using the training sample to obtain a style model of the style domain corresponding to the training sample.

[0151] In one embodiment, the learning rate determination module is specifically used to: calculate the ratio of the number of samples to the preset data amount based on the number of training samples used to train the generative adversarial network model; determine the learning rate corresponding to the training sample based on the ratio and the preset learning rate corresponding to the preset data amount.

[0152] In one embodiment, the target style model determination module 1203 includes an interpolation coefficient determination unit and a target style model determination unit, wherein:

[0153] an interpolation coefficient determination unit, configured to determine an interpolation coefficient, wherein the interpolation coefficient is determined according to a first similarity between the initial style feature and the first style feature, and a second similarity between the initial style feature and the second style feature;

[0154] The target style model determination unit is used to use the interpolation coefficient to interpolate the model parameters of the first style model to obtain the target style model corresponding to the target sample.

[0155] For the specific definition of the style model generation device, please refer to the definition of the style model generation method above, which will not be repeated here. Each module in the above-mentioned style model generation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0156] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.13As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store style modules corresponding to each style domain, including a first style model, a second style model, etc. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a style model generation method is implemented. Those skilled in the art can understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0157] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0158] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0159] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0160] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0161] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A style model generation method, characterized in that: The method comprises: Obtain the initial style features of the target sample; The reference sample is stylized by a first style model to obtain a first style feature corresponding to the first style model, and the reference sample is stylized by a second style model to obtain a second style feature corresponding to the second style model; the initial style feature and the first style feature correspond to image style features in the same style domain; the number of samples trained to obtain the second style model is greater than the number of samples trained to obtain the first style model, and the second style model has the same model structure as the first style model; the reference sample and the target sample are both image samples; the first style feature and the second style feature are image style features of the reference sample corresponding to different style domains; determining an interpolation coefficient, wherein the interpolation coefficient is determined according to a first similarity between the initial style feature and the first style feature, and a second similarity between the initial style feature and the second style feature; The interpolation coefficients are used to interpolate the model parameters of the first style model to obtain a target style model corresponding to the target sample; the target style model is used to stylize the image to obtain a stylized image corresponding to the image and having the image style features of the style domain to which the initial style features belong.

2. The method according to claim 1, characterized in that The target sample includes at least two sample data; the obtaining of the initial style features of the target sample includes: Extracting characteristic information of each sample data in the target sample; The style features corresponding to the feature information are extracted through a preset style extraction network to obtain the initial style features of the target sample.

3. The method according to claim 2, characterized in that After extracting the feature information of each sample data in the target sample and before extracting the style feature corresponding to the feature information through a preset style extraction network, the method further includes: The sub-feature information of the same type in the feature information of each sample data is averaged to obtain the feature information of the target sample.

4. The method according to claim 2, characterized in that: The extracting the style feature corresponding to the feature information through a preset style extraction network to obtain the initial style feature of the target sample includes: Extracting sub-style features corresponding to the feature information of each sample data respectively through a preset style extraction network; Each of the sub-style features is processed to obtain the initial style feature of the target sample.

5. The method according to claim 1, characterized in that The stylizing the reference sample by using the first style model to obtain a first style feature corresponding to the first style model includes: Performing stylization processing on the reference sample by using a first style model to obtain a first stylized result; The style feature of the first stylization result is extracted to obtain the first style feature.

6. The method according to claim 1, characterized in that The first style model and the second style model include a generative adversarial network model, and a method of training to obtain the generative adversarial network model includes: Determining a learning rate corresponding to the training samples according to the number of training samples used to train the generative adversarial network model; Based on the learning rate, the generative adversarial network model is trained using the training samples to obtain a style model of a style domain corresponding to the training samples.

7. The method according to claim 6, characterized in that The determining, according to the number of training samples used to train the generative adversarial network model, a learning rate corresponding to the training samples comprises: Calculate the ratio of the number of training samples to the preset data amount according to the number of training samples used to train the generative adversarial network model; The learning rate corresponding to the training sample is determined according to the ratio and the preset learning rate corresponding to the preset data amount.

8. A style model generation device, characterized in that: The device comprises: An initial style feature acquisition module is used to obtain the initial style features of the target sample; A model style feature acquisition module, used to perform stylized processing on a reference sample through a first style model to obtain a first style feature corresponding to the first style model, and to perform stylized processing on the reference sample through a second style model to obtain a second style feature corresponding to the second style model; the initial style feature and the first style feature correspond to image style features in the same style domain; the number of samples trained to obtain the second style model is greater than the number of samples trained to obtain the first style model, and the model structure of the second style model is the same as that of the first style model; the reference sample and the target sample are both image samples; the first style feature and the second style feature are image style features of the reference sample corresponding to different style domains; A target style model determination module is used to determine an interpolation coefficient, and use the interpolation coefficient to interpolate the model parameters of the first style model to obtain a target style model corresponding to the target sample; the interpolation coefficient is determined according to a first similarity between the initial style feature and the first style feature, and a second similarity between the initial style feature and the second style feature; the target style model is used to stylize the image to obtain a stylized image corresponding to the image and having the image style features of the style domain to which the initial style features belong.

9. The device according to claim 8, characterized in that The target sample includes at least two sample data; the initial style feature acquisition module includes: A characteristic information extraction unit, used to extract characteristic information of each sample data in the target sample; The initial style feature determination unit is used to extract the style feature corresponding to the feature information through a preset style extraction network to obtain the initial style feature of the target sample.

10. The device according to claim 9, characterized in that The initial style feature acquisition module also includes: The feature information processing module is used to average the sub-feature information of the same type in the feature information of each sample data to obtain the feature information of the target sample.

11. The device according to claim 9, characterized in that The initial style feature determination unit comprises: A sub-style feature extraction component, used to extract sub-style features corresponding to the feature information of each sample data through a preset style extraction network; The initial style feature determination component is used to process each of the sub-style features to obtain the initial style feature of the target sample.

12. The device according to claim 8, characterized in that The model style feature acquisition module includes: A first stylized result acquisition unit, configured to perform stylized processing on the reference sample using a first style model to obtain a first stylized result; The first style feature extraction unit is used to extract the style feature of the first stylization result to obtain the first style feature.

13. The device according to claim 8, characterized in that The first style model and the second style model include a generative adversarial network model, and the apparatus further includes: A learning rate determination module, used to determine the learning rate corresponding to the training sample according to the number of training samples used to train the generative adversarial network model; The style model training module is used to perform model training on the generative adversarial network model based on the learning rate and using the training samples to obtain a style model of a style domain corresponding to the training samples.

14. The device according to claim 13, characterized in that The learning rate determination module is specifically used for: Calculate the ratio of the number of training samples to the preset data amount according to the number of training samples used to train the generative adversarial network model; The learning rate corresponding to the training sample is determined according to the ratio and the preset learning rate corresponding to the preset data amount.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images