Image style transfer method, device, equipment and storage medium

Abstract

The present invention discloses an image style transfer method, device, equipment and storage medium, belonging to the field of image processing technology. The present invention obtains a target style degree adjustment parameter and a historical image set; inputs the target style degree adjustment parameter and the historical image set into a preset network model to obtain a reference network model; trains the reference network model based on a target loss function to obtain a target style transfer model; in the inference stage of completing the training, inputs the image to be stylized and the target style degree adjustment parameter into the target style transfer model to obtain a target style image output by the target style transfer model, adds the style degree adjustment parameter during model training, and freely adjusts the stylization degree of the generated stylized result through one-time training, thereby improving the flexibility of adjusting the image stylization degree.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image style migration method, device, equipment and storage medium. Background Art

[0002] Image neural style transfer aims to combine the style of a style image with the content of a content image using a neural network. Most of the early image style transfer methods were based on image features extracted by a classification network trained on a large labeled image dataset, ImageNet. The content of the content image is maintained through the perceptual loss of the content image and the stylized result image at the VGG feature layer, while the style of the stylized result image is guided by the style loss of the Gram matrix of the style image and the stylized result image. However, such VGG network-based methods need to be trained in advance on a large labeled classification image dataset. Collecting such a large labeled classification dataset is very labor-intensive and cumbersome. In addition, since the classification dataset used does not contain the artist's style images, the representation of the classification network trained on it may lead to deviations in style expression.

[0003] In response to the above problems, the current style transfer method is to regard style as a kind of "collective spirit expression". By collecting a set of style images that conform to a certain style, using the training method of generative adversarial networks, the content images are sent to an autoencoder network to convert them into images that follow a certain style through adversarial training. However, compared with the traditional VGG network-based method, the generative adversarial network-based method is difficult to freely adjust its stylization degree, so users need to adjust the training hyperparameters for retraining to change the stylization degree.

[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present invention is to provide an image style transfer method, device, equipment and storage medium, aiming to solve the technical problem that the adjustment of the image stylization degree in the prior art is not flexible enough.

[0006] To achieve the above object, the present invention provides an image style transfer method, which comprises the following steps:

[0007] Obtain the target style degree adjustment parameters and the historical image set;

[0008] Inputting the target style degree adjustment parameter and the historical image set into a preset network model to obtain a reference network model;

[0009] Training the reference network model based on a target loss function to obtain a target style transfer model;

[0010] At the inference stage after completing the training, the image to be stylized and the target style degree adjustment parameter are input into the target style transfer model to obtain the target style image output by the target style transfer model.

[0011] Optionally, obtaining the target style degree adjustment parameter includes:

[0012] Randomly sample from a preset range to obtain a style factor;

[0013] The style factors are mapped through nonlinear mapping to obtain target style degree adjustment parameters.

[0014] Optionally, the step of inputting the target style degree adjustment parameter and the historical image set into a preset network model to obtain a reference network model includes:

[0015] Screening out historical images that do not meet preset conditions from the historical image set;

[0016] Preprocessing the historical images that do not meet the preset conditions to obtain processed historical images;

[0017] The processed historical images and the target style degree adjustment parameters are input into a preset network model to obtain a reference network model.

[0018] Optionally, the processed historical image includes a processed historical content image and a processed historical style image;

[0019] The step of inputting the processed historical image and the target style degree adjustment parameter into a preset network model to obtain a reference network model includes:

[0020] Inputting the processed historical content image into a generator corresponding to a preset network model;

[0021] Generate a reference style image by adjusting parameters of the target style degree and the processed historical content image by the generator;

[0022] The preset network model is trained according to the reference style image and the processed historical style image to obtain a reference network model.

[0023] Optionally, the training the reference network model based on the target loss function to obtain a target style transfer model includes:

[0024] Inputting the reference style image and the processed historical style image into a discriminator corresponding to the reference network model for comparison;

[0025] Training the generator and the discriminator according to the comparison result and the target loss function so that the generator and the discriminator reach a balance;

[0026] When the generator and the discriminator reach a balance, the corresponding reference network model is used as the target style transfer model.

[0027] Optionally, before training the reference network model based on the target loss function to obtain the target style transfer model, the method further includes:

[0028] Obtaining weight parameters of each loss module corresponding to the reference network model;

[0029] Constructing an adversarial loss function, a content loss function, a stylized content loss function, and a style factor loss function according to the weight parameter, the target style degree adjustment parameter, and the historical image set;

[0030] A target loss function is determined according to the anti-loss function, the content loss function, the stylized content loss function and the style factor loss function.

[0031] Optionally, in the inference phase after completing the training, after inputting the image to be stylized and the target style degree adjustment parameter into the target style transfer model to obtain the target style image output by the target style transfer model, the method further includes:

[0032] Obtaining user satisfaction feedback based on the target style image;

[0033] When the satisfaction level does not reach a preset satisfaction level, adjusting the target style degree adjustment parameter;

[0034] The parameters are adjusted based on the adjusted target style degree to obtain a new target style image.

[0035] In addition, to achieve the above-mentioned purpose, the present invention further proposes an image style transfer device, the image style transfer device comprising:

[0036] An acquisition module is used to obtain target style degree adjustment parameters and a historical image set;

[0037] A training module, used for inputting the target style degree adjustment parameter and the historical image set into a preset network model to obtain a reference network model;

[0038] The training module is further used to train the reference network model based on the target loss function to obtain a target style transfer model;

[0039] The processing module is used to input the image to be stylized and the target style degree adjustment parameter into the target style transfer model at the inference stage of completing the training, so as to obtain the target style image output by the target style transfer model.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also proposes an image style transfer device, which includes: a memory, a processor, and an image style transfer program stored in the memory and executable on the processor, wherein the image style transfer program is configured to implement the image style transfer method described above.

[0041] In addition, to achieve the above-mentioned purpose, the present invention further proposes a storage medium, on which an image style transfer program is stored, and when the image style transfer program is executed by a processor, the image style transfer method as described above is implemented.

[0042] The present invention obtains target style degree adjustment parameters and a historical image set; inputs the target style degree adjustment parameters and the historical image set into a preset network model to obtain a reference network model; trains the reference network model based on a target loss function to obtain a target style transfer model; in the inference stage after completing the training, inputs the image to be stylized and the target style degree adjustment parameters into the target style transfer model to obtain a target style image output by the target style transfer model, adds the style degree adjustment parameters during model training, and freely adjusts the stylization degree of the generated stylized result through one-time training, thereby improving the flexibility of adjusting the stylization degree of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural schematic diagram of an image style transfer device in a hardware operating environment involved in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the process of the first embodiment of the image style transfer method of the present invention;

[0045] Figure 3 A schematic diagram of a controllable style transfer system in an embodiment of an image style transfer method of the present invention;

[0046] Figure 4 This is a schematic diagram of the flow chart of the second embodiment of the image style transfer method of the present invention;

[0047] Figure 5 This is a schematic diagram of the flow chart of the third embodiment of the image style transfer method of the present invention;

[0048] Figure 6 FIG. 4 is a structural block diagram of a first embodiment of an image style transfer apparatus according to the present invention.

[0049] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0051] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the structure of an image style transfer device in a hardware operating environment involved in an embodiment of the present invention.

[0052] like Figure 1 As shown, the image style transfer device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0053] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the image style transfer device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0054] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and an image style transfer program.

[0055] exist Figure 1In the image style transfer device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the image style transfer device of the present invention can be set in the image style transfer device, and the image style transfer device calls the image style transfer program stored in the memory 1005 through the processor 1001, and executes the image style transfer method provided by an embodiment of the present invention.

[0056] The embodiment of the present invention provides an image style transfer method, referring to Figure 2 , Figure 2 The figure is a flowchart of a first embodiment of an image style transfer method according to the present invention.

[0057] In this embodiment, the image style transfer method includes the following steps:

[0058] Step S10: Obtain target style degree adjustment parameters and a historical image set.

[0059] In this embodiment, the executor of this embodiment may be an image style transfer device, and the image style transfer device may be an electronic device such as a personal computer, a server, or a vehicle-mounted terminal, or may be other devices or servers that can achieve the same or similar functions. This embodiment is not limited to this. In this embodiment and the following embodiments, the image style transfer method of the present invention is described by taking the image style transfer device as an example.

[0060] It should be noted that, in the current prior art, a style image set of a certain style is trained based on a generative adversarial network to obtain an image style transfer model. After the model is trained, the content image to be style transferred is input into the image style transfer model to convert the content image into an image following a certain style. However, it should be emphasized that although the image style transfer model can convert the input content image into an image following a certain style, based on the different training results, the converted content image follows a certain style but has different degrees of stylization. In actual situations, it may be encountered that the degree of stylization of the converted style image does not meet the user's expectations. The current method requires adjusting the hyperparameters of the generative adversarial network, retraining a new image style transfer model, and then re-inputting the content image into the new image style transfer model to obtain images with different degrees of stylization. In actual operations, it is usually necessary to adjust the image multiple times to obtain an image that meets the user's expectations. In order to obtain generated results with different degrees of stylization, the existing method needs to adjust the hyperparameters of different loss functions in the network model and retrain the model. This method is obviously not flexible enough and has low efficiency. In order to solve the above technical problems, this embodiment proposes a new image style transfer method based on a generative adversarial network, so that the user can freely adjust the style degree of the final stylized image.

[0061] In this embodiment, the image style transfer device of this embodiment is integrated with a controllable style transfer system based on a generative adversarial network, such as Figure 3As shown, the controllable style transfer system adopted in this embodiment includes a content image input module, a style factor adjustment module, a generator stylization module, a style factor injection module, a discriminator module, an adversarial loss module, a content loss module and a stylized content loss module. The content image input module is used to input the content image to be stylized; the generator stylization module is used to generate a stylized result image according to the input content image; the style factor adjustment module is used to randomly sample from a uniform distribution in the interval [0,1] as the style factor input during the training process; the stylization factor injection module maps the style factor samples to the affine transformation parameters of the generator stylization module features through nonlinear mapping, wherein the affine transformation parameters include but are not limited to scale and shift parameters, which are used to exert influence on the generator stylization module to change the stylization degree of the stylized result image. The stylized result image is an image obtained after image style transfer, and the style image set is an image set composed of images following a certain type of style. The discriminator module is used to discriminate and compare the stylized result image with the style images in the style image set. Through continuous discrimination and comparison, the generator stylization module and the discriminator module are balanced so that the stylized result image and the style images in the style image set are visually consistent in style. The discriminator is the structure of the discriminant network. In this embodiment, the discriminator can use a deep convolutional neural network. In practical applications, if the user is not satisfied with the stylization degree of the obtained stylized result image, the stylization degree corresponding to the stylized result image can be adjusted through the style factor adjustment module and the stylization injection module, without adjusting the hyperparameters of different loss functions in the generative adversarial network model and retraining the model. Furthermore, the loss function is composed of an adversarial loss function corresponding to the adversarial loss module, a content loss function corresponding to the content loss module, a stylized content loss function corresponding to the stylized content loss module, and a style factor loss function corresponding to the style factor loss module. In this embodiment, when the stylized result image is obtained, the stylized content loss function corresponding to the stylized content loss module is obtained. Figure 3 After the network model is shown, the network model is further optimized based on the loss function. The loss function can be set accordingly according to actual training requirements, which is not limited in this embodiment.

[0062] In a specific implementation, before performing style transfer and transformation on a stylized image, a corresponding model will be constructed first. In this embodiment, it is necessary to first obtain a target style degree adjustment parameter and a historical image set. The target style degree adjustment parameter is a parameter used to adjust the stylization degree of the stylized result image, which can be used as an affine transformation parameter parameter of the generator stylization module feature. The historical image set includes a historical content image set and a historical style image set. The historical content image set is an image that needs to be style transferred and transformed, and the historical style image set is a style image data set of artworks belonging to the same style.

[0063] Further, the target style degree adjustment parameter in this embodiment is obtained through the style factor. Specifically, in this embodiment, the style factor is mapped by nonlinear mapping to obtain the target style degree adjustment parameter. Nonlinear mapping is also called nonlinear operator. In this embodiment, the Ureson operator, Volterra operator or Hammerstein operator can be used. The style factor can also be mapped in other ways. It can be set accordingly according to actual conditions, which is not limited in this embodiment. The style factor is randomly sampled from a preset range. The preset range can be set to [0,1]. Of course, in this embodiment, the preset range can also be adjusted accordingly according to actual needs, which is not limited in this embodiment.

[0064] Step S20: inputting the target style degree adjustment parameter and the historical image set into a preset network model to obtain a reference network model.

[0065] It should be noted that the preset network model used in this embodiment is a generative adversarial network model. After obtaining the target style degree adjustment parameters and the historical image set, the target style degree adjustment parameters and the historical image set are input into the preset network model to obtain the reference network model. Specifically, the generator in the generative adversarial network model in this embodiment is a generative network structure, which is composed of an encoder-decoder network architecture with a convolutional neural layer as a basic module, which may include multiple jump connection layers, or may be composed of a nearest neighbor upsampling layer and a nearest neighbor downsampling layer and multiple residual blocks. The main function of the generator is to accept the parameters of the style factor and the sample input of the content image input module, and generate a stylized result controlled by the style factor. Therefore, in this embodiment, the target style degree adjustment parameters and the historical image set can be input into the generator to obtain a reference network model.

[0066] Step S30: training the reference network model based on the target loss function to obtain a target style transfer model.

[0067] It should be noted that after obtaining the reference network model, the reference network model is further trained through the target loss function in this embodiment. Through the training of the target loss function, the image style and stylization degree of the stylized result image output by the target style transfer model can be optimized. The trained reference network model is the target style transfer model, which can be used to perform image style transfer transformation on the content image.

[0068] Step S40: After the inference phase of training is completed, the image to be stylized and the target style degree adjustment parameters are input into the target style transfer model to obtain a target style image output by the target style transfer model.

[0069] In a specific implementation, in the inference stage after training is completed, the image to be stylized and the target style degree adjustment parameters are input into the target style transfer model for image style transfer transformation, and then the target style image output by the target style transfer model can be obtained.

[0070] This embodiment obtains a target style degree adjustment parameter and a historical image set; inputs the target style degree adjustment parameter and the historical image set into a preset network model to obtain a reference network model; trains the reference network model based on a target loss function to obtain a target style transfer model; in the inference stage after the training is completed, inputs the image to be stylized and the target style degree adjustment parameter into the target style transfer model to obtain a target style image output by the target style transfer model, adds the style degree adjustment parameter during model training, and freely adjusts the stylization degree of the generated stylized result through one-time training, thereby improving the flexibility of adjusting the stylization degree of the image.

[0071] refer to Figure 4 , Figure 4 The figure is a flow chart of a second embodiment of an image style transfer method according to the present invention.

[0072] Based on the above first embodiment, in the image style transfer method of this embodiment, step S20 specifically includes:

[0073] Step S201: Filter out historical images that do not meet preset conditions from the historical image set.

[0074] It should be noted that there are various images in the acquired historical image set, such as images with different pixels and images of different sizes. In order to improve the accuracy of the model, before the historical images in the historical image set are input into the preset network model, it is necessary to perform image processing on the historical images in the historical image set. In this embodiment, the historical images that need to be processed are screened out from the historical image set by setting preset conditions, that is, historical images that do not meet the preset conditions. Furthermore, in this embodiment, the preset conditions can be set to pixel values ​​or sizes, and historical images in the historical image set that do not meet the preset pixel values ​​or sizes are screened out, and then these historical images are processed. Of course, the preset conditions for screening historical images in this embodiment can be set to other conditions, which can be set accordingly according to actual screening requirements, and are not limited in this embodiment.

[0075] Step S202: pre-processing the historical image that does not meet the preset conditions to obtain a processed historical image.

[0076] In a specific implementation, the historical images that do not meet the preset conditions after screening are preprocessed. The preprocessing method in this embodiment can be an image normalization processing method. Assuming that the preset condition is a pixel value or size, the normalization method can process the historical images that do not meet the conditions into images with the same pixel or size, so that all historical images are images of the same standard. In addition to taking a normalization method to preprocess the historical images that do not meet the preset conditions in this embodiment, other methods can also be taken to preprocess the historical images that do not meet the preset conditions. The specific method can be set accordingly according to the actual image processing requirements, and this is not limited in this embodiment.

[0077] Step S203: inputting the processed historical image and the target style degree adjustment parameter into a preset network model to obtain a reference network model.

[0078] It is easy to understand that, in this embodiment, after pre-processing the historical images in the historical image set, the processed historical images are input into the preset network model to obtain a reference network model.

[0079] In a specific implementation, the historical image set includes historical content images and historical style images. In this embodiment, the processed historical content images and historical style images are input into the generator corresponding to the preset network model, and the target style degree adjustment parameter is also input into the generator. Then, the corresponding reference style image is generated by the generator. The stylization degree corresponding to the reference style image is determined by the target style degree adjustment parameter, and the image style corresponding to the reference style image is determined by the input historical content image. The reference network model in this embodiment is obtained by training the reference style image and the processed historical style image, and the purpose is to ensure that the image style of the reference style image is consistent with the image style of the processed historical style image.

[0080] Furthermore, in order to improve the accuracy and consistency of image style conversion in this embodiment, it can be implemented as follows.

[0081] In a specific implementation, the reference network model also includes a discriminator. After obtaining the reference style image and the processed historical style image, the reference style image and the processed historical style image are input into the discriminator for comparison. The style difference between the reference style image and the processed historical style image can be obtained through the comparison of the discriminator. Then, the generator is adjusted according to the style difference so that the image style of the reference style image finally generated by the generator is consistent with the image style of the processed historical style image. The basis for determining whether the image style of the reference style image finally generated by the generator is consistent with the image style of the processed historical style image is whether the generator and the discriminator have reached a balance. The generator and the discriminator constantly compete with each other to improve the generation ability and the discrimination ability, and finally reach a Nash equilibrium state. When the Nash equilibrium state is reached, it is determined that the image style of the reference style image finally generated by the generator is consistent with the image style of the processed historical style image. At this time, the reference network model is the target style transfer model.

[0082] Furthermore, in this embodiment, when training the generator and the discriminator, a target loss function is also added, and further optimization can be achieved through the target loss function. Specifically, the target loss function in this embodiment is composed of an adversarial loss function, a content loss function, a stylized content loss function, and a style factor loss function, and the adversarial loss function, the content loss function, the stylized content loss function, and the style factor loss function can be constructed according to the weight parameter, the target style degree adjustment parameter, and the historical image set. For example, the adversarial loss function is as follows:

[0083]

[0084] Among them, D, G and F are the discriminator module, the generator stylization module and the style factor injection module respectively, and x and are the corresponding style images in the style training set and the content images in the content training set, respectively, and z is a style factor sample sampled from a uniform distribution in [0,1].

[0085] The content loss function is as follows:

[0086]

[0087] in,‖·‖ 1 represents the L1 norm, λ 1 Represents the weight of the content loss, which is used to adjust the importance of this loss in the overall loss function.

[0088] The stylized content loss function is as follows:

[0089]

[0090] where F(·) represents the perceptual function, which usually represents the features obtained by feeding the image into the bottleneck layer of the encoder in the generator network, and λ 2 Represents the weight of the perceptual loss, which is used to adjust the importance of this loss in the overall loss function.

[0091] Style factor loss function:

[0092]

[0093] For each stylized image generated by the style factor z During training, it is sent to a statistical network T with the same structure as the discriminator network but a different last layer (the last layer of the network outputs the dimension of z) to obtain an estimated value of z: The difference between it and z is calculated as an L2 norm loss. This loss helps to enhance the role of z in the generator network and prevents the generator network from ignoring the style factor z. 3 Represents the weight of the style factor loss function.

[0094] The objective loss function is:

[0095] L total =L adv +L L1 +L percetual +L StyleFactor

[0096] This embodiment filters out historical images that do not meet preset conditions from the historical image set; pre-processes the historical images that do not meet the preset conditions to obtain processed historical images; inputs the processed historical images and the target style degree adjustment parameters into a preset network model to obtain a reference network model, and inputs the reference style image and the processed historical style image into a discriminator corresponding to the reference network model for comparison; trains the generator and the discriminator according to the comparison result and the target loss function to balance the generator and the discriminator; when the generator and the discriminator reach a balance, uses the corresponding reference network model as a target style transfer model, and adds the target style degree adjustment parameters during model training, so that the target style transfer model is more free and flexible when adjusting the image stylization degree.

[0097] refer to Figure 5 , Figure 5 FIG. 4 is a flow chart of a third embodiment of an image style transfer method according to the present invention.

[0098] Based on the above first or second embodiment, a third embodiment of an image style transfer method of the present invention is proposed.

[0099] In this embodiment, after step S40, the following steps are further included:

[0100] Step S50: Obtaining user satisfaction feedback based on the target style image.

[0101] In a specific implementation, after generating the target style image, this embodiment can also display the generated target style image to the user, and then obtain the user's satisfaction feedback based on the generated target style image to determine whether the target style image meets the user's expectations.

[0102] Step S60: When the satisfaction level does not reach a preset satisfaction level, adjusting the target style degree adjustment parameter.

[0103] Step S70: adjusting the parameters based on the adjusted target style degree to obtain a new target style image.

[0104] It is easy to understand that if the satisfaction reaches the preset satisfaction, it means that the generated target style image meets the user's expectations. Otherwise, it means that the generated target style image meets the user's expectations. In this case, the stylization degree of the target style image needs to be adjusted.

[0105] In a specific implementation, when the user is not satisfied with the stylization degree of the target style image, there is no need to retrain the model. Instead, the target style degree adjustment parameters can be adjusted, and then the adjusted target style degree adjustment parameters and the image to be stylized can be re-input into the generator to obtain a new target style image.

[0106] This embodiment obtains user satisfaction based on feedback of the target style image; when the image satisfaction does not reach a preset satisfaction, the target style degree adjustment parameter is adjusted; a new target style image is obtained based on the adjusted target style degree adjustment parameter, and the image stylization degree can be freely adjusted by adjusting the target style degree adjustment parameter, thereby improving the flexibility of image stylization degree adjustment.

[0107] In addition, an embodiment of the present invention further provides a storage medium, on which an image style transfer program is stored. When the image style transfer program is executed by a processor, the steps of the image style transfer method described above are implemented.

[0108] Since the storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here.

[0109] Reference Figure 6 , Figure 6 FIG. 4 is a structural block diagram of a first embodiment of an image style transfer apparatus according to the present invention.

[0110] like Figure 6 As shown, the image style transfer device proposed in the embodiment of the present invention includes:

[0111] The acquisition module 10 is used to obtain the target style adjustment parameters and the historical image set.

[0112] The training module 20 is used to input the target style degree adjustment parameter and the historical image set into a preset network model to obtain a reference network model.

[0113] The training module 20 is further used to train the reference network model based on a target loss function to obtain a target style transfer model.

[0114] The processing module 30 is used to input the image to be stylized and the target style degree adjustment parameter into the target style transfer model in the inference stage of completing the training, so as to obtain the target style image output by the target style transfer model.

[0115] This embodiment obtains a target style degree adjustment parameter and a historical image set; inputs the target style degree adjustment parameter and the historical image set into a preset network model to obtain a reference network model; trains the reference network model based on a target loss function to obtain a target style transfer model; in the inference stage after the training is completed, inputs the image to be stylized and the target style degree adjustment parameter into the target style transfer model to obtain a target style image output by the target style transfer model, adds the style degree adjustment parameter during model training, and freely adjusts the stylization degree of the generated stylized result through one-time training, thereby improving the flexibility of adjusting the stylization degree of the image.

[0116] In one embodiment, the acquisition module 10 is further used to perform random sampling from a preset range to obtain a style factor; and map the style factor through nonlinear mapping to obtain a target style degree adjustment parameter.

[0117] In one embodiment, the training module 20 is also used to filter out historical images that do not meet preset conditions from the historical image set; pre-process the historical images that do not meet the preset conditions to obtain processed historical images; and input the processed historical images and the target style degree adjustment parameters into a preset network model to obtain a reference network model.

[0118] In one embodiment, the processed historical image includes a processed historical content image and a processed historical style image;

[0119] The training module 20 is also used to input the processed historical content image into the generator corresponding to the preset network model; adjust the parameters according to the target style degree through the generator, and generate a reference style image based on the input processed historical content image; train the preset network model according to the reference style image and the processed historical style image to obtain a reference network model.

[0120] In one embodiment, the training module 20 is further used to input the reference style image and the processed historical style image into the discriminator corresponding to the reference network model for comparison; train the generator and the discriminator according to the comparison and target loss function to balance the generator and the discriminator; when the generator and the discriminator reach a balance, use the corresponding reference network model as the target style transfer model.

[0121] In one embodiment, the image style transfer apparatus further comprises: determining a construction module;

[0122] The construction module is also used to obtain weight parameters of each loss module corresponding to the reference network model; construct an adversarial loss function, a content loss function, a stylized content loss function and a style factor loss function according to the weight parameters, the target style degree adjustment parameters and the historical image set; and determine a target loss function according to the adversarial loss function, the content loss function, the stylized content loss function and the style factor loss function.

[0123] In one embodiment, the image style transfer apparatus further includes: an adjustment module;

[0124] The adjustment module is also used to obtain user satisfaction based on the target style image feedback; when the image satisfaction does not reach a preset satisfaction, adjust the target style degree adjustment parameter; obtain a new target style transfer model based on the adjusted target style degree adjustment parameter, and obtain a new target style image through the new target style transfer model.

[0125] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.

[0126] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0127] In addition, for technical details not fully described in this embodiment, reference can be made to the image style transfer method provided in any embodiment of the present invention, and will not be repeated here.

[0128] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0129] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0131] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for image style transfer, characterized in that: The image style transfer method comprises: Obtain the target style degree adjustment parameters and the historical image set; Inputting the target style degree adjustment parameter and the historical image set into a preset network model to obtain a reference network model; Training the reference network model based on a target loss function to obtain a target style transfer model; In the inference phase after completing the training, the image to be stylized and the target style degree adjustment parameter are input into the target style transfer model to obtain a target style image output by the target style transfer model; Before training the reference network model based on the target loss function to obtain the target style transfer model, the method further includes: Obtaining weight parameters of each loss module corresponding to the reference network model; Constructing an adversarial loss function, a content loss function, a stylized content loss function, and a style factor loss function according to the weight parameter, the target style degree adjustment parameter, and the historical image set; Determine a target loss function according to the anti-loss function, the content loss function, the stylized content loss function and the style factor loss function; The reference network model includes a generator and a discriminator. After obtaining the reference style image and the processed historical style image, the reference style image and the processed historical style image are input into the discriminator for comparison. The style difference between the reference style image and the processed historical style image is obtained through the comparison of the discriminator. Then, the generator is adjusted according to the style difference so that the image style of the reference style image finally generated by the generator is consistent with the image style of the processed historical style image. When the Nash equilibrium state is reached, it is determined that the image style of the reference style image finally generated by the generator is consistent with the image style of the processed historical style image, and the reference network model is determined to be a target style transfer model. Among them, the adversarial loss function is L adv =argmax D argmin G E x~p(x) log[D(x)]-E z~p(z),x~p(x) log[1-D(G(z, ))] D, G and F are the discriminator module, generator stylization module and style factor injection module respectively, x and are the corresponding style images in the style training set and the content images in the content training set, respectively. z is a style factor sample sampled from a uniform distribution of [0, 1]. The content loss function is , Among them, ‖·‖1 represents the L1 norm, λ1 represents the weight of the image content loss, The stylized content loss function is Where F(·) represents the perceptual function, λ2 represents the weight of the perceptual loss, The style factor loss function is Among them, z is the style factor, is the generated stylized image, T is the statistical network, λ3 represents the weight of the style factor loss function, The target loss function is L total =L adv +L L1 +L percetual +L StyleFactor .

2. The image style transfer method according to claim 1, characterized in that: The step of obtaining the target style degree adjustment parameter includes: Randomly sample from a preset range to obtain a style factor; The style factors are mapped through nonlinear mapping to obtain target style degree adjustment parameters.

3. The image style transfer method according to claim 1, characterized in that: The step of inputting the target style degree adjustment parameter and the historical image set into a preset network model to obtain a reference network model includes: Screening out historical images that do not meet preset conditions from the historical image set; Preprocessing the historical images that do not meet the preset conditions to obtain processed historical images; The processed historical images and the target style degree adjustment parameters are input into a preset network model to obtain a reference network model.

4. The image style transfer method according to any one of claims 1 to 3, characterized in that: In the inference phase of completing the training, after inputting the image to be stylized and the target style degree adjustment parameter into the target style transfer model to obtain the target style image output by the target style transfer model, the method further includes: Obtaining user satisfaction feedback based on the target style image; When the satisfaction level does not reach a preset satisfaction level, adjusting the target style degree adjustment parameter; The parameters are adjusted based on the adjusted target style degree to obtain a new target style image.

5. An image style transfer device, characterized in that: The image style transfer device comprises: An acquisition module is used to obtain target style degree adjustment parameters and a historical image set; A training module, used for inputting the target style degree adjustment parameter and the historical image set into a preset network model to obtain a reference network model; The training module is further used to train the reference network model based on the target loss function to obtain a target style transfer model; A processing module, configured to input the image to be stylized and the target style degree adjustment parameter into the target style transfer model at the inference stage of completing the training, so as to obtain a target style image output by the target style transfer model; The training module is also used to obtain weight parameters of each loss module corresponding to the reference network model; construct an adversarial loss function, a content loss function, a stylized content loss function and a style factor loss function according to the weight parameters, the target style degree adjustment parameters and the historical image set; determine the target loss function according to the anti-loss function, the content loss function, the stylized content loss function and the style factor loss function, wherein the reference network model includes a generator and a discriminator, after obtaining the reference style image and the processed historical style image, the reference style image and the processed historical style image are input into the discriminator for comparison, and the style difference between the reference style image and the processed historical style image is obtained by comparison of the discriminator, and then the generator is adjusted according to the style difference, so that the image style of the reference style image finally generated by the generator is consistent with the image style of the processed historical style image, and when the Nash equilibrium state is reached, it is determined that the image style of the reference style image finally generated by the generator is consistent with the image style of the processed historical style image, and the reference network model is determined to be a target style transfer model, wherein the adversarial loss function is L adv =argmax D argmin G E x~p(x) log[D(x)]-E z~p(z),x~p(x) log[1-D(G(z, ))] D, G and F are the discriminator module, generator stylization module and style factor injection module respectively, x and are the corresponding style images in the style training set and the content images in the content training set, respectively. z is a style factor sample sampled from a uniform distribution of [0, 1]. The content loss function is , Among them, ‖·‖1 represents the L1 norm, λ1 represents the weight of the image content loss, The stylized content loss function is Where F(·) represents the perceptual function, λ2 represents the weight of the perceptual loss, The style factor loss function is Among them, z is the style factor, is the generated stylized image, T is the statistical network, λ3 represents the weight of the style factor loss function, The target loss function is L total =L adv +L L1 +L percetual +L StyleFactor .

6. An image style transfer device, characterized in that: The image style transfer device comprises: a memory, a processor, and an image style transfer program stored in the memory and executable on the processor, wherein the image style transfer program is configured to implement the image style transfer method according to any one of claims 1 to 4.

7. A storage medium, characterized in that: An image style transfer program is stored on the storage medium, and when the image style transfer program is executed by the processor, the image style transfer method according to any one of claims 1 to 4 is implemented.

Images