Image style conversion model training method, image style conversion method and device

By training the image style conversion model for different target styles, the problem of the style conversion model in the existing technology supports single style conversion, and the flexibility and diversity of style conversion are achieved.

CN113850712BActive Publication Date: 2024-11-26BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111034624.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-03
Publication Date
2024-11-26
Estimated Expiration
2041-09-03

AI Technical Summary

Technical Problem

The existing image style conversion model can only support the conversion of a single style, lacks flexibility, and cannot effectively realize the switching between different styles and the conversion of different styles in the image area.

Method used

By obtaining the style parameters used to train the preset model, including single style, fusion multiple styles and partitioned multiple styles, training the preset model for each target style, generating an image style conversion model, realizing switching between different styles and within the image area Different styles of conversion.

Benefits of technology

It realizes the flexibility of the image style conversion model, and can realize different types of style conversion effects according to the input style parameters, solving the problem of a single style conversion method.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113850712B_ABST
    Figure CN113850712B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a training method, an image style conversion method and a device for an image style conversion model. The training method for the image style conversion model includes: obtaining style parameters used for training a preset model; wherein the style parameters are used to represent parameters of a target style, and the preset model is used to convert a sample original image into a target style based on the style parameters, and the target style is one of the following three types: single style, fusion of multiple styles or partitioned multiple styles; obtaining a training target image after converting the sample original image according to the style parameters; for each type of target style, according to the sample original image, the style parameters of the corresponding type of target style, and the training target image obtained after conversion according to the corresponding style parameters, for each type of target style, the preset model is trained to obtain an image style conversion model, which solves the problem of a single style conversion method supported by the image style conversion model in the related art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a training method for an image style conversion model, an image style conversion method and a device. Background Art

[0002] Style conversion, also known as style migration, refers to the process of transforming digital images through computers (or other intelligent computing devices) to output images with styles, such as oil paintings, fine brush paintings, hand-painted portraits and other common styles. A common implementation method in related technologies is that one model supports one style, and different style conversion models are trained for different styles. This style conversion model only supports the conversion of a single style, and the style conversion method is not flexible enough. Summary of the invention

[0003] The present disclosure provides a training method, an image style conversion method and an apparatus for an image style conversion model, so as to at least solve the problem that the image style conversion model in the related art supports a single style conversion method. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a method for training an image style transfer model is provided, comprising:

[0005] Get the style parameters used for training the preset model; wherein the style parameters are used to represent the parameters of the target style, and the preset model is used to convert the sample original image into the target style based on the style parameters, wherein the target style is one of the following three types: single style, fusion of multiple styles, or partitioned multi-style, where the single style is a single type of style, the fusion of multiple styles is a style that combines the effects of multiple types of styles, and the partitioned multi-style is that different areas of the converted image correspond to different types of styles;

[0006] Obtaining a training target image after converting the sample original image according to the style parameters; wherein the training target image is used as a training target of an output image of a preset model;

[0007] For each type of target style, a preset model is trained according to the sample original image, the style parameters of the corresponding type of target style, and the training target image obtained after conversion according to the corresponding style parameters;

[0008] After the preset model is trained for each target style, an image style transfer model is obtained.

[0009] Optionally, the step of obtaining style parameters used for training a preset model includes:

[0010] Select the target style for this training from a variety of target styles;

[0011] When the target style is a single style, a style type of the single style is selected to obtain style parameters; wherein the style parameters of the single style include the style type of the single style;

[0012] When the target style is a fusion of multiple styles, style types of multiple styles are selected, and a fusion weight of each style type is obtained to obtain style parameters; wherein the style parameters of the fusion of multiple styles include the style types of multiple styles and a fusion weight of each style type;

[0013] When the target style is partitioned multi-style, style types of multiple styles are selected, and the partitions corresponding to each style type are obtained to obtain style parameters; wherein the style parameters of the partitioned multi-style include style types of multiple styles and the partitions corresponding to each style type.

[0014] Optionally, when the target style is a single style, a style type of the single style is selected, and the step of obtaining the style parameter includes: randomly selecting a style code from a plurality of candidate style codes; wherein each style code is used to represent a corresponding style type;

[0015] In the case where the target style is a fusion of multiple styles, the style types of multiple styles are selected, and the fusion weight of each style type is obtained. The step of obtaining the style parameter includes: randomly selecting P style codes from multiple candidate style codes, and randomly generating P values ​​whose sum is 1, the P values ​​are respectively used as the fusion weights of the P style codes, P is a positive integer greater than or equal to 2, and each value is greater than 0 and less than 1;

[0016] In the case where the target style is partitioned multi-style, style types of multiple styles are selected, and the partitions corresponding to each style type are obtained. The steps of obtaining style parameters include: randomly selecting M style codes from multiple candidate style codes, and randomly generating M binary matrices, the M binary matrices are respectively used as partition mask templates for the M style codes, wherein M is a positive integer greater than or equal to 2, the matrix sum of the M binary matrices is an all-1 matrix, and the matrix size of each binary matrix is ​​the same as the pixel matrix size of the original image of the sample.

[0017] Optionally, obtaining a training target image after converting the sample original image according to the style parameters includes:

[0018] When the target style is a single style, obtain the style image corresponding to the sample original image and the single type of style to obtain the training target image;

[0019] When the target style is a fusion of multiple styles, the sample original image and style images corresponding to multiple styles are obtained to obtain multiple style images. According to the fusion weight of each style type, the multiple style images are weighted summed according to the corresponding fusion weight to obtain the training target image.

[0020] When the target style is partitioned multi-style, the sample original image and style images corresponding to multiple styles are obtained to obtain multiple style images, and the training target image is calculated by combining the partitions corresponding to each style type.

[0021] According to a second aspect of an embodiment of the present disclosure, there is provided an image style conversion method, including:

[0022] Obtaining an image to be converted and style parameters; wherein the style parameters are used to represent parameters of the target style;

[0023] The image to be converted and the style parameters are input into the image style conversion model to obtain a target image after the image to be converted is converted into the target style; wherein the image style conversion model is trained according to the training method of the image style conversion model of the first aspect.

[0024] According to a third aspect of an embodiment of the present disclosure, a training device for an image style transfer model is provided, comprising:

[0025] A first acquisition unit is used to acquire style parameters used for training a preset model; wherein the style parameters are used to represent parameters of a target style, and the preset model is used to convert a sample original image into a target style based on the style parameters, wherein the target style is one of the following three types: a single style, a fusion of multiple styles, or a partitioned multi-style, wherein the single style is a style of a single type, the fusion of multiple styles is a style that combines effects of multiple types of styles, and the partitioned multi-style is a style in which different areas of the converted image correspond to different types of styles;

[0026] A second acquisition unit is used to acquire a training target image after converting the sample original image according to the style parameters; wherein the training target image is used as a training target of an output image of a preset model;

[0027] A first training unit is used to train a preset model for each type of target style according to a sample original image, a style parameter of the target style of the corresponding type, and a training target image obtained after conversion according to the corresponding style parameter;

[0028] The second training unit is used to obtain an image style conversion model after the preset model is trained for each target style.

[0029] Optionally, the first acquiring unit includes:

[0030] A first selection subunit is used to select a target style for this training from multiple target styles;

[0031] The second selection subunit is used to select a style type of the single style to obtain style parameters when the target style is a single style; wherein the style parameters of the single style include the style type of the single style;

[0032] The third selection subunit is used to select style types of multiple styles when the target style is a fusion of multiple styles, and obtain a fusion weight of each style type to obtain a style parameter; wherein the style parameter of the fusion of multiple styles includes the style types of multiple styles and the fusion weight of each style type;

[0033] The fourth selection subunit is used to select style types of multiple styles when the target style is partitioned multi-style, and obtain the partition corresponding to each style type to obtain style parameters; wherein the style parameters of the partitioned multi-style include style types of multiple styles and the partition corresponding to each style type.

[0034] Optionally, the second selection subunit is further used to randomly select a style code from a plurality of candidate style codes when the target style is a single style; wherein each style code is used to represent a corresponding style type;

[0035] The third selection subunit is further used for randomly selecting P style codes from multiple candidate style codes when the target style is a fusion of multiple styles, and randomly generating P values ​​whose sum is 1, the P values ​​are respectively used as fusion weights of the P style codes, P is a positive integer greater than or equal to 2, and each value is greater than 0 and less than 1;

[0036] The fourth selection subunit is also used to randomly select M style codes from multiple candidate style codes when the target style is partitioned multi-style, and randomly generate M binary matrices. The M binary matrices are respectively used as partition mask templates for the M style codes, where M is a positive integer greater than or equal to 2, the sum of the matrices of the M binary matrices is an all-1 matrix, and the matrix size of each binary matrix is ​​the same as the pixel matrix size of the original image of the sample.

[0037] Optionally, the second acquiring unit includes:

[0038] The first acquisition subunit is used to acquire the sample original image and the style image corresponding to the single type of style to obtain the training target image when the target style is a single style;

[0039] The second acquisition subunit is used to acquire the sample original image and style images corresponding to multiple styles respectively when the target style is a fusion of multiple styles, to obtain multiple style images, and to perform weighted summation of the multiple style images according to the corresponding fusion weights according to the fusion weights of each style type, to obtain a training target image;

[0040] The third acquisition subunit is used to obtain the sample original image and the style images corresponding to multiple styles when the target style is partitioned multi-style, obtain multiple style images, and combine the partitions corresponding to each style type to calculate the training target image.

[0041] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image style conversion device, including:

[0042] An acquisition unit, used to acquire an image to be converted and style parameters; wherein the style parameters are used to represent parameters of a target style;

[0043] An input unit is used to input the image to be converted and the style parameters into the image style conversion model to obtain a target image after the image to be converted is converted into a target style; wherein the image style conversion model is trained according to the training method of the image style conversion model of the first aspect.

[0044] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0045] processor;

[0046] a memory for storing processor-executable instructions;

[0047] The processor is configured to execute instructions to implement a training method for an image style transfer model as provided in the first aspect and any one of its examples, or an image style transfer method as provided in the second aspect and any one of its examples.

[0048] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to execute a training method for an image style transfer model as provided in the first aspect and any one of its examples, or an image style transfer method as provided in the second aspect and any one of its examples.

[0049] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements a training method for an image style transfer model as provided in the first aspect and any one of its examples, or an image style transfer method as provided in the second aspect and any one of its examples.

[0050] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0051] By training the preset model for different target styles, the image style transfer model obtained after training can achieve different style conversion effects, including switching between different single styles, converting to a fusion of multiple styles, and converting different types of styles in different areas of the image. Therefore, different types of style conversion effects can be achieved according to different input style parameters. The style conversion method is flexible and changeable, which solves the problem of a single style conversion method supported by the image style transfer model in related technologies.

[0052] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0054] Figure 1 The present invention is a flowchart of a method for training an image style transfer model according to an exemplary embodiment.

[0055] Figure 2 It is a schematic diagram showing conversion effects of different styles according to an exemplary embodiment.

[0056] Figure 3 It is a schematic diagram showing a conversion effect of multi-style fusion conversion according to an exemplary embodiment.

[0057] Figure 4 The diagram is a schematic diagram showing a conversion effect of a partition conversion according to an exemplary embodiment.

[0058] Figure 5 is a schematic diagram of a multi-style transfer model according to an exemplary embodiment.

[0059] Figure 6 The figure is a flowchart of a method for image style conversion according to an exemplary embodiment.

[0060] Figure 7 The figure is a block diagram of a training device for an image style transfer model according to an exemplary embodiment.

[0061] Figure 8 The figure is a block diagram of a device for image style conversion according to an exemplary embodiment.

[0062] Fig. 9It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0063] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0064] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0065] Figure 1 is a flowchart of a method for training an image style transfer model according to an exemplary embodiment. Figure 1 As shown, the following steps may be specifically included.

[0066] Step 101, obtaining style parameters used for training a preset model.

[0067] In the disclosed embodiment, the style parameter is one of the inputs of the preset model during training and the image style transfer model obtained after training the preset model. The structure of the preset model and the image style transfer model can use the structure of the neural network model in the related art, such as a feedforward neural network model, a recurrent neural network model, a symmetrical connection neural network model, etc. After training the preset model, the image style transfer model is obtained.

[0068] The above style parameters are used to represent the parameters of the target style. The preset model is used to convert the sample original image into the target style based on the style parameters.

[0069] The target style is one of the following three categories: single style, fusion of multiple styles or partitioned multiple styles. The preset model / image style transfer model can achieve at least three target style conversions. Specifically, the model can convert the input image into a target style of the type indicated by the input style parameter.

[0070] Specifically, single style refers to a single type of style, fusion multi-style refers to a style that combines the effects of multiple types of styles, and partitioned multi-style refers to different types of styles corresponding to different areas in the converted image.

[0071] In this way, the step 101 of obtaining the style parameters used for training the preset model can be to select the target style of this training from multiple target styles, and then, according to different target styles, execute corresponding steps.

[0072] When the target style is a single style, the style type of the single style can be selected to obtain the style parameters; wherein the style parameters of the single style include the style type of the single style. When the target style is a single style, the preset model can switch between different single styles, that is, inputting an original image can output style images of different styles. The "switch" for switching is the style parameter. The style parameters corresponding to the single style are used to represent the corresponding style type, and each type of style can be represented by different style codes.

[0073] For example, Figure 2 As shown, for an original image 200, after being processed by a preset model, different style images can be generated according to different input style parameters. The style images 201 to 204 can correspond to the style parameters: 00, 01, 10, 11, respectively; or, the style parameters can also be encoded using a one-hot encoding method. For example, the style images 201 to 204 can correspond to the style parameters: 0001, 0010, 0100, 1000, respectively.

[0074] When the target style is a fusion of multiple styles, you can select style types of multiple styles, and obtain the fusion weight of each style type to obtain style parameters; wherein the style parameters of the fusion of multiple styles include style types of multiple styles and the fusion weight of each style type. When the target style is a fusion of multiple styles, the preset model can fuse different styles, that is, convert the image into a fusion style obtained after the fusion of multiple styles. The style parameters corresponding to the fusion of multiple styles can represent each fused style and the weight of each style. For example, it can be represented in the form of a multi-dimensional array, where each dimension of the array represents the weight ratio of the corresponding type of style in the fused style, the weight ratio is between 0 and 1, and the weights of all styles add up to 1.

[0075] For example, a preset model can support Figure 2 There are four style types shown. Then, when performing multi-style fusion, at least two types of styles can be selected from these four style types for fusion, and the style types selected for fusion and the weight of each style type can be described by style parameters.

[0076] Exemplarily, the style parameter may be [0, i, j, 0], which indicates that the weights of the first and fourth style types are 0 and are not selected for fusion, while the second and third style types are selected for fusion, with weights i and j respectively, both of which are in the interval (0, 1), and i+j=1.

[0077] Figure 3 The diagram illustrates the effect of fusion of multiple styles when the weights of the second and third style types are different. The sample original image 300 can obtain style image 301 when converted to the second style type, and obtain style image 302 when converted to the third style type. Style images 3001 to 3004 are the effects of fusion of the second style type and the third style type, wherein the weight of the second style type in style images 3001 to 3004 becomes smaller and smaller, and the weight of the third style type becomes larger and larger.

[0078] When the target style is partitioned multi-style, you can select multiple style types, and obtain the partitions corresponding to each style type to obtain style parameters; wherein the style parameters of the partitioned multi-style include the style types of multiple styles and the partitions corresponding to each style type. When the target style is partitioned multi-style, the preset model can transfer different style types to different regions, that is, the style type of each region in the generated style image is different. It should be noted that the fineness of the partition can be pixel level. The style parameters of the partitioned multi-style are used to represent each style type and the image area corresponding to each style type.

[0079] In one example, the style parameters can be represented by a mask matrix of the same size as the digital matrix of the image, where each element in the mask matrix is ​​used to represent the style used at the corresponding pixel position. Figure 4 The figure shows the effect of converting the left and right partitions into different styles respectively.

[0080] From the above description, it can be seen that the style parameters can represent the target style and specific parameters of the desired preset model / image conversion style model. By presetting different parameter formats for style parameters of different target styles, the model (including the preset model and the image conversion style model) can identify the target style of the desired conversion according to the parameter format.

[0081] In order to train the preset model, style parameters can be randomly generated within the parameter value range of the style parameters according to the parameter format of the style parameters. In this way, the flexibility of the model training process can be improved, the sample space can be increased, and the style transfer effect can be improved.

[0082] Step 102, obtaining a training target image after converting the sample original image according to the style parameters.

[0083] The training target image is used as the training target of the output image of the preset model. When training the preset model, the preset model can obtain the output style image according to the input style parameters and the sample original image, and then the internal parameters of the preset model can be adjusted according to the gap between the training target image and the output style image. In this way, by using different sample original images or style parameters for multiple iterative training, the gap between the output style image and the training target image can be reduced.

[0084] Optionally, when the target style is a single style, a style image corresponding to the sample original image and a single type of style may be obtained to obtain a training target image.

[0085] When the target style is a fusion of multiple styles, the style images corresponding to the sample original image and multiple styles can be obtained to obtain multiple style images. Then, according to the fusion weight of each style type, the multiple style images are weighted and summed according to the corresponding fusion weights to obtain the training target image.

[0086] When the target style is partitioned multi-style, the style images corresponding to the sample original image and multiple styles can be obtained to obtain multiple style images, and the training target image can be calculated by combining the partitions corresponding to each style type.

[0087] Optionally, when executing step 102, a training target image can be generated based on the samples in the image sample set. The image sample set includes the sample original image, and the style image of each different style type converted from the sample original image, for example, oil painting style, sketch style, comic style, realistic style, etc. The above style types are only used for illustrative examples and are not used to limit the style types of the embodiments of the present disclosure. In this way, after obtaining the style parameters, a sample original image can be obtained, and then, a style image of the corresponding style type can be obtained according to the style parameters, and a training target image can be generated according to the style parameters. Through this implementation, different training target images can be generated according to different style parameters, which increases the sample space for training and improves the efficiency of using style images in the samples.

[0088] Step 103 , for each type of target style, a preset model is trained according to the sample original image, the style parameters of the corresponding type of target style, and the training target image obtained after conversion according to the corresponding style parameters.

[0089] When training a preset model, a sample original image and corresponding style parameters can be input into the preset model for each training. After the preset model obtains an output image, the model parameters in the preset model can be adjusted according to the difference between the output image and the training target image. Through multiple iterative trainings, the difference between the output image and the training target image can be reduced, so that the image output by the preset model can be more consistent with the target style indicated by the style parameters.

[0090] The gap between the output image and the training target image can be specifically evaluated by parameter indicators such as the mean square error of all pixels and the mean square error of each region, which is not limited in the embodiments of the present disclosure. When adjusting the model parameters in the preset model according to the gap, a parameter adjustment algorithm of deep learning in the relevant technology can be used, for example, a parameter adjustment algorithm based on Bayesian optimization, a parameter adjustment algorithm based on gradient descent, etc.

[0091] Step 104: After the preset model is trained for each target style, an image style conversion model is obtained.

[0092] The structure of the preset model / image style conversion model in the embodiments of the present disclosure may include: Figure 5 The input encoder, latent space and output encoder are shown in three parts, wherein the style parameters can be added to any one or more of the above parts, that is, the style parameters can be added to the three parts simultaneously or separately during training. Specifically, the preset model / image style conversion model can be a neural network model, and adding the style parameters is equivalent to adding an additional encoding channel for representing the style parameters to the convolution layer output of the corresponding part. Preferably, the encoding channel of the style parameters can be added to the latent space and the output encoder.

[0093] The disclosed embodiment trains the preset model for different target styles, so that the image style conversion model obtained after training can achieve conversion effects of different styles, including switching between different single styles, converting to a fusion of multiple styles, and converting different types of styles in different areas of the image. Therefore, it is possible to achieve conversion effects of different types of styles according to different input style parameters, and the style conversion method is flexible and changeable, which solves the problem of a single style conversion method supported by the image style conversion model in the related art. The image style conversion model obtained by training can achieve the effect of conversion to different target styles, which can reduce the storage space and running memory required by the model, and solves the problem of large storage space and running memory required to load multiple models when realizing image processing effects of multiple styles in the related art.

[0094] In one example, when the target style is a single style, the step of selecting a style type of the single style and obtaining style parameters may specifically be to randomly select a style code from a plurality of candidate style codes, wherein each style code is used to represent a corresponding style type.

[0095] In this way, when executing step 102 to obtain the training target image, the style image corresponding to the sample original image and the style coding can be selected.

[0096] In the case where the target style is a fusion of multiple styles, the above-mentioned step of selecting style types of multiple styles, obtaining the fusion weight of each style type, and obtaining the style parameter may be to randomly select P style codes from multiple candidate style codes, and randomly generate P values ​​whose sum is 1. Here, the P values ​​are respectively used as the fusion weights of the P style codes, P is a positive integer greater than or equal to 2, and each value is greater than 0 and less than 1.

[0097] In this way, when executing step 102 to obtain the training target image, the style image corresponding to each style code of the sample original image can be obtained according to the style parameter, and the image after the fusion of multiple styles can be further calculated according to the weight formula. The specific method can be to obtain the pixel value matrix of P style images corresponding to the sample original image, multiply each matrix by the weight of the corresponding style type, and sum them up. The formula can be referred to:

[0098] G=∑k f T f (f=1,……,k),∑k f =1,

[0099] Among them, G is the training target image, k f is the weight parameter of the f-th style type, T f is the pixel value matrix of the style image corresponding to the f-th style type. For example, taking the fusion of two styles as an example, G=kT1+(1-k)T2.

[0100] In the case where the target style is a partitioned multi-style, the above-mentioned step of selecting style types of multiple styles, obtaining the partition corresponding to each style type, and obtaining the style parameters can be to randomly select M style codes from multiple candidate style codes and randomly generate M binary matrices. Here, the M binary matrices are respectively used as partition mask templates of the M style codes, where M is a positive integer greater than or equal to 2, the matrix sum of the M binary matrices is an all-1 matrix, and the matrix size of each binary matrix is ​​the same as the pixel matrix size of the sample original image.

[0101] Thus, when executing step 102 to obtain the training target image, the style image corresponding to each style code of the sample original image can be obtained according to the style parameter, and the training target image can be further calculated according to the partition mask template. For example, the training target image can be calculated by the following formula:

[0102] G=∑K f (T f )(f=1,……,k),

[0103] Among them, G is the training target image, K f is the partition mask template of the f-th style type (a matrix), T f K is the pixel value matrix of the style image corresponding to the f-th style type. f With T f The matrices are of the same size, that is, the matrices are all a×b in size.

[0104] K f () can be regarded as a mask function of the f-th style type, which can be a binary matrix with element values ​​0 or 1, used to describe whether each pixel uses the f-th style type. If the f-th style is used, the pixel value of the corresponding pixel in the corresponding style image is retained. Otherwise, if the f-th style type is not used, the pixel value of the corresponding pixel is 0, and the pixel positions retained by different style types do not overlap.

[0105] Exemplarily, the partition mask template K1 of the first style type is: The partition mask template K2 of the second style type is: The pixel value matrix T1 of the sample original image corresponding to the style image of the first style type is: The pixel value matrix T2 of the sample original image corresponding to the style image of the second style type is: Correspondingly, K1(T1) get

[0106] That is, when it is necessary to realize partition conversion into different styles, the image area (pixel value position) corresponding to each style type can be determined according to the style parameters, the pixel value of the corresponding part can be extracted from the corresponding style image, and the pixel values ​​extracted from all style types can be combined into an image according to the corresponding pixel positions to obtain the training target image.

[0107] Through the specific implementation method of generating style parameters described above, style parameters that are more in line with the characteristics of different target styles can be obtained, so that the model obtained after training can obtain images that are more in line with the corresponding target style when converting the image style.

[0108] The present disclosure also provides an image style conversion method. Figure 6 As shown, the following steps are included:

[0109] Step 501: Obtain the image to be converted and style parameters.

[0110] The style parameters are used to represent the parameters of the target style. The style parameters can be input or selected by the user. The image to be converted is the image to be input into the image style conversion model, and specifically can be an image provided or selected by the user.

[0111] In an example application scenario, the image style conversion method provided by the embodiment of the present disclosure can be applied to the client, the image style conversion model is trained by the server through the training method of the image style conversion model provided by the embodiment of the present disclosure, the client can be installed in the terminal, and the server can be configured in a server connected to the terminal for communication. The client can receive the image to be converted selected and uploaded by the user in the album through the terminal, and receive the user's selection of a style parameter from multiple style parameters provided by the client, or the user can input a value that conforms to the preset parameter format and preset parameter range of the style parameter.

[0112] Step 502: input the image to be converted and the style parameters into an image style conversion model to obtain a target image after converting the image to be converted into a target style.

[0113] Among them, the image style transfer model is trained according to the image style transfer model training method provided in the embodiment of the present disclosure.

[0114] The image style conversion method provided in the embodiment of the present disclosure trains a preset model for different target styles, so that the image style conversion model obtained after training can achieve different style The conversion effect includes switching between different single styles, converting to a fusion of multiple styles, and converting different types of styles in different areas of the image. Therefore, different types of style conversion effects can be achieved according to different input style parameters. The style conversion method is flexible and changeable, which solves the problem of a single style conversion method supported by the image style conversion model in related technologies.

[0115] The embodiments of the present disclosure also provide a training device for an image style transfer model, which can be used to execute the training method for the image style transfer model provided by the embodiments of the present disclosure. For the descriptions not described in detail in the training device for the image style transfer model provided by the embodiments of the present disclosure, reference can be made to the relevant descriptions of the above-mentioned embodiments of the training method for the image style transfer model, and no further details will be given here.

[0116] Figure 7is a block diagram of a training device for an image style transfer model according to an exemplary embodiment. Figure 7 The device includes a first acquisition unit 601, a second acquisition unit 602, a first training unit 603 and a second training unit 604.

[0117] The first acquisition unit 601 is used to acquire style parameters used for training the preset model; wherein the style parameters are used to represent parameters of the target style, and the preset model is used to convert the sample original image into the target style based on the style parameters, wherein the target style is one of the following three types: single style, fusion of multiple styles, or partitioned multi-style, where the single style is a single type of style, the fusion of multiple styles is a style that combines multiple types of style effects, and the partitioned multi-style is that different areas of the converted image correspond to different types of styles;

[0118] The second acquisition unit 602 is used to acquire a training target image after converting the sample original image according to the style parameters; wherein the training target image is used as a training target of the output image of the preset model;

[0119] The first training unit 603 is used to train a preset model for each type of target style according to the sample original image, the style parameters of the corresponding type of target style, and the training target image obtained after conversion according to the corresponding style parameters;

[0120] The second training unit 604 is used to obtain an image style conversion model after the preset model is trained for each target style.

[0121] Optionally, the first acquiring unit 601 may include:

[0122] A first selection subunit is used to select a target style for this training from multiple target styles;

[0123] The second selection subunit is used to select a style type of the single style to obtain style parameters when the target style is a single style; wherein the style parameters of the single style include the style type of the single style;

[0124] The third selection subunit is used to select style types of multiple styles when the target style is a fusion of multiple styles, and obtain a fusion weight of each style type to obtain a style parameter; wherein the style parameter of the fusion of multiple styles includes the style types of multiple styles and the fusion weight of each style type;

[0125] The fourth selection subunit is used to select style types of multiple styles when the target style is partitioned multi-style, and obtain the partition corresponding to each style type to obtain style parameters; wherein the style parameters of the partitioned multi-style include style types of multiple styles and the partition corresponding to each style type.

[0126] Optionally, the second selection subunit may also be used to randomly select a style code from a plurality of candidate style codes when the target style is a single style; wherein each style code is used to represent a corresponding style type;

[0127] The third selection subunit may also be used to randomly select P style codes from multiple candidate style codes when the target style is a fusion of multiple styles, and randomly generate P values ​​whose sum is 1, the P values ​​are respectively used as fusion weights of the P style codes, P is a positive integer greater than or equal to 2, and each value is greater than 0 and less than 1;

[0128] The fourth selection subunit can also be used to randomly select M style codes from multiple candidate style codes when the target style is a partitioned multi-style, and randomly generate M binary matrices, and the M binary matrices are respectively used as partition mask templates for the M style codes, where M is a positive integer greater than or equal to 2, the sum of the matrices of the M binary matrices is an all-1 matrix, and the matrix size of each binary matrix is ​​the same as the pixel matrix size of the original image of the sample.

[0129] Optionally, the second obtaining unit 602 may include:

[0130] The first acquisition subunit is used to acquire the sample original image and the style image corresponding to the single type of style to obtain the training target image when the target style is a single style;

[0131] The second acquisition subunit is used to acquire the sample original image and style images corresponding to multiple styles respectively when the target style is a fusion of multiple styles, to obtain multiple style images, and to perform weighted summation of the multiple style images according to the corresponding fusion weights according to the fusion weights of each style type, to obtain a training target image;

[0132] The third acquisition subunit is used to obtain the sample original image and the style images corresponding to multiple styles when the target style is partitioned multi-style, obtain multiple style images, and combine the partitions corresponding to each style type to calculate the training target image.

[0133] The training device of the image style transfer model provided by the embodiment of the present disclosure trains the preset model for different target styles, so that the image style transfer model obtained after training can achieve different style conversion effects, including switching between different single styles, converting to a fusion of multiple styles, and converting different types of styles in different areas of the image. Therefore, different types of style conversion effects can be achieved according to different input style parameters, and the style conversion method is flexible and changeable, which solves the problem of a single style conversion method supported by the image style transfer model in the related art.

[0134] The embodiments of the present disclosure also provide an image style conversion device, which can be used to execute the image style conversion method provided by the embodiments of the present disclosure. For the descriptions not described in detail in the image style conversion device provided by the embodiments of the present disclosure, reference can be made to the relevant descriptions of the above-mentioned embodiments of the image style conversion method, and no further details will be given here.

[0135] Figure 8 is a block diagram of an image style conversion device according to an exemplary embodiment. Figure 8 , the device includes an acquisition unit 701 and an input unit 702.

[0136] The acquisition unit 701 is used to acquire the image to be converted and style parameters; wherein the style parameters are used to represent parameters of the target style.

[0137] The input unit 702 is used to input the image to be converted and the style parameters into the image style transfer model to obtain a target image after the image to be converted is converted into the target style; wherein the image style transfer model is trained according to the training method of the image style transfer model of the embodiment of the present disclosure.

[0138] The image style conversion device provided by the embodiment of the present disclosure trains a preset model for different target styles, so that the image style conversion model obtained after training can achieve conversion effects of different styles, including switching between different single styles, converting to a fusion of multiple styles, and converting different types of styles in different areas of the image. Therefore, different types of style conversion effects can be achieved according to different input style parameters. The style conversion method is flexible and changeable, which solves the problem of a single style conversion method supported by the image style conversion model in the related art.

[0139] Regarding the device in the above embodiment, the specific manner in which each module performs operations and the technical effects of the solution have been described in detail in the embodiment of the method, and will not be elaborated here.

[0140] The present disclosure also provides an electronic device, specifically in combination with Fig. 9 Provide detailed explanation.

[0141] Fig. 9 1 is a structural block diagram of an electronic device 900 according to an exemplary embodiment. The electronic device 900 can implement the training method of the image style conversion model or the image style conversion method according to the embodiment of the present disclosure.

[0142] In an exemplary embodiment, the present disclosure embodiment further provides a computer-readable storage medium including instructions. For example, the computer-readable storage medium may be a memory 902 including instructions in the electronic device 900. The instructions may be executed by the processor 901 in the electronic device 900 to enable the electronic device 900 to complete the training method of the image style conversion model or the image style conversion method provided in the present disclosure embodiment. Alternatively, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0143] In an exemplary embodiment, the present disclosure also provides a computer program product, including a computer program / instructions, which can be executed by the processor 901 to implement the image style transfer model training method or image style transfer method provided by the present disclosure.

[0144] Specifically, the processor 901 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present disclosure.

[0145] The memory 902 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 902 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 902 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 902 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 902 is a non-volatile solid-state memory. In a particular embodiment, the memory 902 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0146] The processor 901 may read and execute computer program instructions stored in the memory 902 to perform the following steps:

[0147] Get the style parameters used for training the preset model; wherein the style parameters are used to represent the parameters of the target style, and the preset model is used to convert the sample original image into the target style based on the style parameters, wherein the target style is one of the following three types: single style, fusion of multiple styles, or partitioned multi-style, where the single style is a single type of style, the fusion of multiple styles is a style that combines the effects of multiple types of styles, and the partitioned multi-style is that different areas of the converted image correspond to different types of styles;

[0148] Obtaining a training target image after converting the sample original image according to the style parameters; wherein the training target image is used as a training target of an output image of a preset model;

[0149] For each type of target style, a preset model is trained according to the sample original image, the style parameters of the corresponding type of target style, and the training target image obtained after conversion according to the corresponding style parameters;

[0150] After the preset model is trained for each target style, an image style transfer model is obtained.

[0151] In an optional implementation, the processor 901 executes the step of acquiring the style parameters used for training the preset model, which may include the following steps:

[0152] Select the target style for this training from a variety of target styles;

[0153] When the target style is a single style, a style type of the single style is selected to obtain style parameters; wherein the style parameters of the single style include the style type of the single style;

[0154] When the target style is a fusion of multiple styles, style types of multiple styles are selected, and a fusion weight of each style type is obtained to obtain style parameters; wherein the style parameters of the fusion of multiple styles include the style types of multiple styles and a fusion weight of each style type;

[0155] When the target style is partitioned multi-style, style types of multiple styles are selected, and the partitions corresponding to each style type are obtained to obtain style parameters; wherein the style parameters of the partitioned multi-style include style types of multiple styles and the partitions corresponding to each style type.

[0156] In an optional implementation, when the target style is a single style, the processor 901 performs the step of selecting a style type of a single style, and obtaining the style parameter includes: randomly selecting a style code from a plurality of candidate style codes; wherein each style code is used to represent a corresponding style type;

[0157] In the case where the target style is a fusion of multiple styles, the processor 901 executes the steps of selecting style types of multiple styles and obtaining the fusion weight of each style type, and obtaining the style parameter includes: randomly selecting P style codes from multiple candidate style codes, and randomly generating P values ​​whose sum is 1, the P values ​​are respectively used as the fusion weights of the P style codes, P is a positive integer greater than or equal to 2, and each value is greater than 0 and less than 1;

[0158] When the target style is a partitioned multi-style, the processor 901 executes the selection of style types of multiple styles and obtains the partition corresponding to each style type. The step of obtaining style parameters includes: randomly selecting M style codes from multiple candidate style codes, and randomly generating M binary matrices, and the M binary matrices are respectively used as partition mask templates for the M style codes, wherein M is a positive integer greater than or equal to 2, the matrix sum of the M binary matrices is an all-1 matrix, and the matrix size of each binary matrix is ​​the same as the pixel matrix size of the sample original image.

[0159] Optionally, the processor 901 executes the step of acquiring the training target image after converting the sample original image according to the style parameters, which may include:

[0160] When the target style is a single style, obtain the style image corresponding to the sample original image and the single type of style to obtain the training target image;

[0161] When the target style is a fusion of multiple styles, the sample original image and style images corresponding to multiple styles are obtained to obtain multiple style images. According to the fusion weight of each style type, the multiple style images are weighted summed according to the corresponding fusion weight to obtain the training target image.

[0162] When the target style is partitioned multi-style, the sample original image and style images corresponding to multiple styles are obtained to obtain multiple style images, and the training target image is calculated by combining the partitions corresponding to each style type.

[0163] Or, in another example, the processor 901 is configured to perform the following steps:

[0164] Obtaining an image to be converted and style parameters; wherein the style parameters are used to represent parameters of the target style;

[0165] The image to be converted and the style parameters are input into the image style conversion model to obtain a target image after the image to be converted is converted into the target style; wherein the image style conversion model is trained according to the training method of the image style conversion model of the embodiment of the present disclosure.

[0166] In one example, the 900 may also include a transceiver 903 and a bus 904. Fig. 9 As shown, the processor 901, the memory 902 and the transceiver 903 are connected via a bus 904 and communicate with each other.

[0167] Bus 904 includes hardware, software or both. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral control interconnect (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. Where appropriate, bus 904 may include one or more buses. Although the present disclosure embodiment describes and shows a specific bus, the present disclosure considers any suitable bus or interconnection.

[0168] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0169] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an image style transfer model, characterized in that: include: Obtain style parameters used for training a preset model; wherein the style parameters are used to represent parameters of a target style, and the preset model is used to convert a sample original image into the target style based on the style parameters, wherein the target style is one of the following three types: a single style, a fusion of multiple styles, or a partitioned multi-style, wherein the single style is a style of a single type, the fusion of multiple styles is a style that combines multiple types of style effects, and the partitioned multi-style is a style in which different areas of the converted image correspond to different types of styles; Acquire a training target image after converting the sample original image according to the style parameters; wherein the training target image is used as a training target of the output image of the preset model; For each type of the target style, the preset model is trained according to the sample original image, the style parameters of the target style of the corresponding type, and the training target image obtained after conversion according to the corresponding style parameters; After the preset model is trained for each target style, an image style conversion model is obtained; The step of obtaining the style parameters used for training the preset model includes: Selecting the target style for this training from among the multiple target styles; In the case where the target style is the single style, randomly selecting a style code from a plurality of candidate style codes; wherein each style code is used to represent a corresponding style type; In the case where the target style is the fused multiple styles, randomly selecting P style codes from a plurality of candidate style codes, and randomly generating P values ​​whose sum is 1, the P values ​​are respectively used as fusion weights of the P style codes, P is a positive integer greater than or equal to 2, and each of the values ​​is greater than 0 and less than 1; When the target style is the partitioned multi-style, M style codes are randomly selected from multiple candidate style codes, and M binary matrices are randomly generated. The M binary matrices are respectively used as partition mask templates for the M style codes, wherein M is a positive integer greater than or equal to 2, the matrix sum of the M binary matrices is an all-1 matrix, and the matrix size of each of the binary matrices is the same as the pixel matrix size of the original image of the sample.

2. The method for training an image style transfer model according to claim 1, characterized in that: The step of obtaining a training target image after converting the sample original image according to the style parameters includes: When the target style is the single style, obtaining the style image corresponding to the sample original image and the single type of style to obtain the training target image; When the target style is the fused multiple styles, obtaining the style images corresponding to the sample original image and the multiple styles respectively to obtain multiple style images, and performing weighted summation of the multiple style images according to the corresponding fusion weights according to the fusion weights of each style type to obtain the training target image; When the target style is the partitioned multi-style, the style images corresponding to the sample original image and the multiple styles are obtained to obtain multiple style images, and the training target image is calculated by combining the partitions corresponding to each style type.

3. A method for image style transfer, characterized in that: include: Acquire an image to be converted and style parameters; wherein the style parameters are used to represent parameters of a target style; The image to be converted and the style parameters are input into an image style conversion model to obtain a target image after the image to be converted is converted into the target style; wherein the image style conversion model is trained according to the training method of the image style conversion model according to any one of claims 1-2.

4. A training device for an image style transfer model, characterized in that: include: A first acquisition unit is used to acquire style parameters used for training a preset model; wherein the style parameters are used to represent parameters of a target style, and the preset model is used to convert a sample original image into the target style based on the style parameters, wherein the target style is one of the following three types: a single style, a fusion of multiple styles, or a partitioned multi-style, wherein the single style is a style of a single type, the fusion of multiple styles is a style that combines multiple types of style effects, and the partitioned multi-style is a style in which different areas of the converted image correspond to different types of styles; A second acquisition unit is used to acquire a training target image after converting the sample original image according to the style parameters; wherein the training target image is used as a training target of the output image of the preset model; A first training unit is used to train the preset model for each type of the target style according to the sample original image, the style parameters of the target style of the corresponding type, and the training target image obtained after conversion according to the corresponding style parameters; A second training unit, configured to obtain an image style conversion model after the preset model is trained for each target style; Wherein, the first acquisition unit includes: A first selection unit, configured to select the target style for this training from among the multiple target styles; a second selection unit, configured to randomly select a style code from a plurality of candidate style codes when the target style is the single style; wherein each style code is used to represent a corresponding style type; a third selection unit, configured to randomly select P style codes from a plurality of candidate style codes when the target style is the fused multi-style, and randomly generate P values ​​whose sum is 1, the P values ​​being respectively used as fusion weights of the P style codes, P being a positive integer greater than or equal to 2, and each of the values ​​being greater than 0 and less than 1; The fourth selection unit is used to randomly select M style codes from multiple candidate style codes and randomly generate M binary matrices when the target style is the partitioned multi-style. The M binary matrices are respectively used as partition mask templates for the M style codes, wherein M is a positive integer greater than or equal to 2, the matrix sum of the M binary matrices is an all-1 matrix, and the matrix size of each of the binary matrices is the same as the pixel matrix size of the original image of the sample.

5. The training device for the image style transfer model according to claim 4, characterized in that: The second acquiring unit includes: A first acquisition subunit is used to acquire, when the target style is the single style, the style image corresponding to the sample original image and the single type of style to obtain the training target image; A second acquisition subunit is used for acquiring, when the target style is the fused multiple styles, the style images corresponding to the sample original image and the multiple styles respectively, to obtain a plurality of style images, and according to the fusion weight of each style type, weighted summing the plurality of style images according to the corresponding fusion weights to obtain the training target image; The third acquisition subunit is used to obtain the style images corresponding to the sample original image and the multiple styles respectively when the target style is the partitioned multi-style, obtain multiple style images, and calculate the training target image in combination with the partition corresponding to each style type.

6. An image style conversion device, characterized in that: include: An acquisition unit, used to acquire an image to be converted and style parameters; wherein the style parameters are used to represent parameters of a target style; An input unit is used to input the image to be converted and the style parameters into an image style conversion model to obtain a target image after the image to be converted is converted into the target style; wherein the image style conversion model is trained according to the training method of the image style conversion model according to any one of claims 1-2.

7. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 3.

8. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 2.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.