Model generation method, stylized image generation method, device and electronic equipment

Abstract

The present disclosure relates to a model generation method, a stylized image generation method, an apparatus and an electronic device, and relates to the technical field of computers. The method encodes sample content images and sample style images with the same image content based on an encoding layer in an image coding and decoding model, to obtain sample features corresponding to the sample content images and the sample style images respectively. The sample features corresponding to the sample style images are fused with random noise data based on a to-be-trained stylization model, to generate sample noise features. Predicted noise data is generated based on the sample noise features and sample features corresponding to the sample content images, and the to-be-trained stylization model is trained based on the predicted noise data and the random noise data until a first stop condition is reached, the training is ended, and the current to-be-trained stylization model is taken as a stylization processing model. The stylization processing model is used to convert a to-be-processed image into a stylized image conforming to a preset style. The convenience of obtaining a stylized image can be improved to a certain extent.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model generation method, a stylized image generation method, an apparatus, and an electronic device. Background Technology

[0002] Currently, image processing methods are becoming increasingly diverse. Among them, the demand for image stylization is growing. Stylization, also known as style transfer, is essentially using a reference stylistic image to represent a content image; that is, transferring the style of an artistic image to a regular content image. Stylizing an image allows the content image to acquire additional artistic styles while retaining its original content, such as illustration, cartoon, comic, oil painting, or ink painting styles.

[0003] Therefore, how to easily obtain stylized images has become a technical problem that urgently needs to be solved. Summary of the Invention

[0004] This disclosure provides a model generation method, a stylized image generation method, an apparatus, and an electronic device to solve the problem of how to conveniently obtain stylized images. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a model generation method is provided, comprising:

[0006] Based on the encoding layer in the image encoding and decoding model, sample content maps and sample style maps with the same image content are encoded separately to obtain sample features corresponding to the sample content map and the sample style map respectively; the sample content map and the sample style map have different styles, and the sample style map is an image with a preset style;

[0007] Based on the stylization model to be trained, the sample features corresponding to the sample style map are fused with random noise data to generate sample noise features;

[0008] Based on the sample noise features and the sample features corresponding to the sample content map, predictive noise data is generated, and the stylization model to be trained is trained based on the predictive noise data and the random noise data until a first stopping condition is reached, the training ends, and the current stylization model to be trained is used as the stylization processing model; the stylization processing model is used to convert the image to be processed into a stylized image that conforms to the preset style.

[0009] Optionally, generating predicted noise data based on the sample noise features and the sample features corresponding to the sample content map includes:

[0010] The sample noise features, the first preset time, and the sample features corresponding to the sample content map are input into the noise prediction layer of the stylization model to be trained to obtain the predicted noise data corresponding to the first preset time.

[0011] The step of training the stylization model based on the predicted noise data and the random noise data includes:

[0012] The parameters of the stylization model to be trained are adjusted based on the predicted noise data and the random noise data.

[0013] Optionally, inputting the sample noise features, the first preset time point, and the sample features corresponding to the sample content map into the noise prediction layer of the stylization model to be trained includes:

[0014] The first preset time point is vectorized to obtain time features, and the sample noise features and the sample features corresponding to the sample content map are spliced ​​together to obtain spliced ​​features.

[0015] The splicing features and the temporal features are input into the noise prediction layer of the stylization model to be trained.

[0016] Optionally, the method further includes:

[0017] A preset sample image is used as the input to the encoding and decoding model to be trained, and the preset sample image is downsampled by the encoding layer in the encoding and decoding model to be trained at a specified ratio to obtain the hidden state features of the preset sample image in the hidden space.

[0018] The hidden state features are decoded by the decoding layer in the codec model to be trained, and the reconstructed image corresponding to the preset sample image is obtained.

[0019] Based on the preset sample image and the reconstructed image, the parameters of the codec model to be trained are adjusted until the second stopping condition is met, the training ends, and the current codec model to be trained is used as the image codec model.

[0020] Optionally, adjusting the parameters of the codec model to be trained based on the preset sample image and the reconstructed image includes:

[0021] Determine the absolute pixel error between the preset sample image and the reconstructed image, the content error between the preset sample image and the reconstructed image, and the difference parameters between the reconstructed image and the preset sample image;

[0022] The model loss of the codec model to be trained is determined based on the absolute pixel error, the content error, and the difference parameter.

[0023] The parameters of the codec model to be trained are adjusted based on the model loss.

[0024] According to a second aspect of the present disclosure, a stylized image generation method is provided, comprising:

[0025] The image to be processed is encoded by the encoding layer in the image encoding and decoding model to obtain a first feature; the first feature is used to characterize the image features of the image to be processed.

[0026] The first feature is fused with the noise data through a stylized processing model to obtain the second feature, and the predicted noise data is determined based on the second feature;

[0027] The second feature is gradually denoised based on the predicted noise data using the stylization processing model to obtain the third feature.

[0028] The third feature is decoded by the decoding layer in the image encoding and decoding model to obtain a stylized image corresponding to the image to be processed; the stylized image has the same image content as the image to be processed, and the image content of the stylized image conforms to a preset style.

[0029] Optionally, the step of fusing the first feature with the noise data through the stylization processing model to obtain the second feature, and determining the predicted noise data based on the second feature, includes:

[0030] The first feature is fused with random noise data at a second preset time to obtain the second feature;

[0031] The second feature, the second preset time, and the first feature are input into the noise prediction layer of the stylization processing model to obtain the predicted noise data corresponding to the second preset time.

[0032] The step of progressively denoising the second feature based on the predicted noise data using the stylization processing model to obtain the third feature includes:

[0033] Based on the predicted noise data, a denoising operation is performed on the current second feature to obtain the updated second feature;

[0034] If the denoising stop condition is not met, the updated second feature is denoised again based on the updated second feature, the next time after the second preset time, and the first feature, until the denoising stop condition is met.

[0035] According to a third aspect of the present disclosure, a model generation apparatus is provided, comprising:

[0036] The first encoding module is configured to perform encoding on sample content maps and sample style maps with the same image content based on the encoding layer in the image encoding and decoding model, respectively, to obtain sample features corresponding to the sample content map and the sample style map; the sample content map and the sample style map have different styles, and the sample style map is an image with a preset style;

[0037] The generation module is configured to perform the fusion of sample features corresponding to the sample style map with random noise data based on the style model to be trained, so as to generate sample noise features;

[0038] The first training module is configured to generate predicted noise data based on the sample noise features and the sample features corresponding to the sample content map, and train the stylization model to be trained based on the predicted noise data and the random noise data until a first stopping condition is met, thereby ending the training and using the current stylization model to be trained as the stylization processing model; the stylization processing model is used to convert the image to be processed into a stylized image that conforms to the preset style.

[0039] Optionally, the first training module is specifically configured to execute:

[0040] The sample noise features, the first preset time, and the sample features corresponding to the sample content map are input into the noise prediction layer of the stylization model to be trained to obtain the predicted noise data corresponding to the first preset time.

[0041] The parameters of the stylization model to be trained are adjusted based on the predicted noise data and the random noise data.

[0042] Optionally, the first training module is further configured to execute:

[0043] The first preset time point is vectorized to obtain time features, and the sample noise features and the sample features corresponding to the sample content map are spliced ​​together to obtain spliced ​​features.

[0044] The splicing features and the temporal features are input into the noise prediction layer of the stylization model to be trained.

[0045] Optionally, the device further includes:

[0046] The downsampling module is configured to take a preset sample image as input to the codec model to be trained, and downsample the preset sample image by a specified ratio through the encoding layer in the codec model to be trained, so as to obtain the latent state features of the preset sample image in the latent space.

[0047] The first decoding module is configured to decode the hidden state features through the decoding layer in the codec model to be trained, so as to obtain the reconstructed image corresponding to the preset sample image;

[0048] The adjustment module is configured to perform parameter adjustment on the codec model to be trained based on the preset sample image and the reconstructed image until a second stopping condition is met, thereby ending the training and using the current codec model to be trained as the image codec model.

[0049] Optionally, the adjustment module is specifically configured to execute:

[0050] Determine the absolute pixel error between the preset sample image and the reconstructed image, the content error between the preset sample image and the reconstructed image, and the difference parameters between the reconstructed image and the preset sample image;

[0051] The model loss of the codec model to be trained is determined based on the absolute pixel error, the content error, and the difference parameter.

[0052] The parameters of the codec model to be trained are adjusted based on the model loss.

[0053] According to a fourth aspect of the present disclosure, a stylized image generation apparatus is provided, comprising:

[0054] The encoding module is configured to encode the image to be processed through the encoding layer in the image encoding / decoding model to obtain a first feature; the first feature is used to characterize the image features of the image to be processed.

[0055] The determination module is configured to perform a process of fusing the first feature with the noise data through a stylization processing model to obtain a second feature, and to determine the predicted noise data based on the second feature;

[0056] The denoising module is configured to perform stepwise denoising of the second feature based on the predicted noise data using the stylization processing model to obtain the third feature;

[0057] The second decoding module is configured to decode the third feature through the decoding layer in the image encoding and decoding model to obtain a stylized image corresponding to the image to be processed; the stylized image has the same image content as the image to be processed, and the image content of the stylized image conforms to a preset style.

[0058] Optionally, the determining module is specifically configured to execute:

[0059] The first feature is fused with random noise data at a second preset time to obtain the second feature;

[0060] The second feature, the second preset time, and the first feature are input into the noise prediction layer of the stylization processing model to obtain the predicted noise data corresponding to the second preset time.

[0061] The noise reduction module is specifically configured to execute:

[0062] Based on the predicted noise data, a denoising operation is performed on the current second feature to obtain the updated second feature;

[0063] If the denoising stop condition is not met, the updated second feature is denoised again based on the updated second feature, the next time after the second preset time, and the first feature, until the denoising stop condition is met.

[0064] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0065] processor;

[0066] Memory used to store the processor's executable instructions;

[0067] The processor is configured to execute the instructions to implement the method as described in any one of the first aspects.

[0068] According to a fifth aspect of the present disclosure, a storage medium is provided such that, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device performs the method as described in any one of the first aspects.

[0069] According to a sixth aspect of the present disclosure, a computer program product is provided, the computer program product including readable program instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the method as described in any one of the first aspects.

[0070] The technical solutions provided by the embodiments of this disclosure offer at least the following beneficial effects: In these embodiments, based on the encoding layer in the image encoding and decoding model, sample content maps and sample style maps with the same image content are encoded separately to obtain sample features corresponding to each sample content map and sample style map. The sample content map and sample style map have different styles, and the sample style map is an image with a preset style. Next, based on the stylization model to be trained, the sample features corresponding to the sample style map are fused with random noise data to generate sample noise features. Finally, based on the sample noise features and the sample features corresponding to the sample content map, prediction noise data is generated, and the stylization model to be trained is trained based on the prediction noise data and random noise data until a first stopping condition is met, training ends, and the current stylization model to be trained is used as the stylization processing model. The stylization processing model is used to convert the image to be processed into a stylized image that conforms to the preset style. In this way, the stylization processing model obtained through training can directly convert the image to be processed into a stylized image that conforms to the preset style, thus improving the convenience of obtaining stylized images to a certain extent. Furthermore, by converting sample images into sample features and training the stylization model in the form of sample features, the efficiency of training the stylization model can be improved to some extent, since the amount of data in the features is often less than the amount of data in the image itself.

[0071] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0072] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0073] Figure 1 This is a flowchart illustrating a model generation method according to an exemplary embodiment;

[0074] Figure 2 This is a flowchart illustrating a stylized image generation method according to an exemplary embodiment.

[0075] Figure 3 This is a schematic diagram illustrating a processing procedure according to an exemplary embodiment;

[0076] Figure 4 This is a schematic diagram illustrating a training process according to an exemplary embodiment;

[0077] Figure 5 This is a schematic diagram illustrating another processing flow according to an exemplary embodiment;

[0078] Figure 6This is a block diagram illustrating a model generation apparatus according to an exemplary embodiment;

[0079] Figure 7 This is a block diagram illustrating a stylized image generation apparatus according to an exemplary embodiment;

[0080] Figure 8 This is a block diagram illustrating an apparatus for model generation according to an exemplary embodiment;

[0081] Figure 9 This is a block diagram illustrating another apparatus for model generation according to an exemplary embodiment. Detailed Implementation

[0082] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0083] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0084] Figure 1 This is a flowchart illustrating a model generation method according to an exemplary embodiment, such as... Figure 1 As shown, the method may include the following steps:

[0085] Step 101: Based on the encoding layer in the image encoding and decoding model, the sample content map and sample style map with the same image content are encoded respectively to obtain the sample features corresponding to the sample content map and the sample style map respectively; the sample content map and the sample style map have different styles, and the sample style map is an image with a preset style.

[0086] Step 102: Based on the stylization model to be trained, fuse the sample features corresponding to the sample style map with random noise data to generate sample noise features.

[0087] Step 103: Generate predicted noise data based on the sample noise features and the sample features corresponding to the sample content map, and train the stylization model to be trained based on the predicted noise data and the random noise data until the first stopping condition is reached, end the training, and use the current stylization model to be trained as the stylization processing model; the stylization processing model is used to convert the image to be processed into a stylized image that conforms to the preset style.

[0088] In this embodiment, the image encoding / decoding model can be pre-trained and includes an encoder and a decoder. The encoder in the image encoding / decoding model can be used to convert the image into a feature representation in a specified space to reduce the amount of data during subsequent stylization model processing. Sample content maps and sample style maps can be paired; in this embodiment, multiple pairs of sample content map-sample style map can be prepared for use. For any pair of sample content map-sample style map, the sample content map and sample style map in the pair can be input into the image encoding / decoding model respectively, and then the output of the encoder in the image encoding / decoding model is obtained to obtain the sample features corresponding to the sample content map and the sample features corresponding to the sample style map. Further, Gaussian noise with the same shape as the sample features corresponding to the sample style map can be randomly generated as random noise data for this round of training and added to the sample features corresponding to the sample style map to achieve fusion and obtain sample noise features. In this embodiment, predicted noise data is further generated based on the sample noise features and the sample features corresponding to the sample content map, and the stylization model to be trained is trained based on the predicted noise data and the random noise data. In this way, the predicted noise data output by the stylization model to be trained can get closer and closer to the real random noise data as the training process progresses, which can, to a certain extent, ensure the accuracy of the stylized image generated when the model actually processes the image to be processed.

[0089] The first stopping condition can be a pre-selected training stopping condition. For example, the first stopping condition could be that the number of training epochs reaches a preset epoch threshold, or that the model loss value of the stylization model to be trained is less than a preset threshold, and so on. Since the stylization processing model is trained based on sample content maps and sample style maps with the same image content, and the image content of the sample style maps conforms to the preset style, the stylization processing model, after training, can learn the ability to convert an image into a preset style. Therefore, the image to be processed can ultimately be converted into a stylized image that conforms to the preset style. The preset style can be set according to actual needs; for example, the preset style can be an illustration style, watercolor style, oil painting style, cartoon style, etc. Image content can refer to the objects included in the image, such as the subject, background, etc.

[0090] In summary, the model generation method provided in this embodiment first encodes sample content maps and sample style maps with the same image content based on the encoding layer in the image encoding and decoding model, respectively, to obtain sample features corresponding to the sample content maps and sample style maps. The sample content maps and sample style maps have different styles, and the sample style map is an image with a preset style. Next, based on the stylization model to be trained, the sample features corresponding to the sample style map are fused with random noise data to generate sample noise features. Finally, prediction noise data is generated based on the sample noise features and the sample features corresponding to the sample content maps, and the stylization model to be trained is trained based on the prediction noise data and random noise data until a first stopping condition is met, at which point the training ends and the current stylization model to be trained is used as the stylization processing model. The stylization processing model is used to convert the image to be processed into a stylized image that conforms to the preset style. In this way, the stylization processing model obtained through training can directly convert the image to be processed into a stylized image that conforms to the preset style, thus improving the convenience of obtaining stylized images to a certain extent. Furthermore, by converting sample images into sample features and training the stylization model in the form of sample features, the efficiency of training the stylization model can be improved to some extent, since the amount of data in the features is often less than the amount of data in the image itself.

[0091] Figure 2 This is a flowchart illustrating a stylized image generation method according to an exemplary embodiment, such as... Figure 2 As shown, the method may include the following steps:

[0092] Step 201: Encode the image to be processed through the encoding layer in the image encoding and decoding model to obtain a first feature; the first feature is used to characterize the image features of the image to be processed.

[0093] Step 202: The first feature is fused with the noise data through a stylization processing model to obtain the second feature, and the predicted noise data is determined based on the second feature.

[0094] Step 203: The second feature is gradually denoised using the stylization processing model based on the predicted noise data to obtain the third feature.

[0095] Step 204: Decode the third feature through the decoding layer in the image encoding and decoding model to obtain the stylized image corresponding to the image to be processed; the stylized image has the same image content as the image to be processed, and the image content of the stylized image conforms to a preset style.

[0096] In this embodiment, the data volume of the first feature is smaller than the data volume of the image to be processed. The stylization processing model is trained based on sample content maps and sample style maps with the same image content, and the image content of the sample style maps conforms to a preset style. This stylization processing model can be generated based on the above-described pattern generation method.

[0097] The image to be processed can be a content image that needs to be stylized. The first feature can be a feature representation that transforms the image to be processed into a specified space. The stylization processing model can be a pre-trained model for image stylization. The image encoding / decoding model can be used to transform the image into a feature representation in the specified space to reduce the amount of data processed by the subsequent stylization processing model. Transforming the image into a feature representation in the specified space can ignore some redundant information and noise in the original image space, so that the transformed feature representation focuses on the most critical features, thereby making inference in the specified space more efficient, that is, improving the processing efficiency of the subsequent stylization processing model. The first feature can also be understood as the extracted image features, which are essentially some compressed image information. At the same time, the image encoding / decoding model can be used to restore the feature representation to the image to ensure that the final output can be in the form of an image.

[0098] Stylization processing models can be used to stylize images. Specifically, the model adds noise data to a first feature of the image to be processed; that is, it fuses the first feature with the noise data to obtain a second feature. The added noise data can be Gaussian noise, and obtaining the second feature is equivalent to turning the image into random noise data by adding Gaussian noise. The second feature is then progressively denoised based on the predicted noise data to obtain a third feature. Specifically, this process of progressively denoising until a third feature representing a stylized image is generated. The predicted noise data can be generated by the stylization processing model based on the second feature. Correspondingly, the decoding layer in the image encoding / decoding model decodes the third feature to obtain the stylized image corresponding to the image to be processed.

[0099] Since the stylization model is trained on sample content maps and sample style maps with the same image content, and the image content of the sample style maps conforms to the preset style, the stylization model can learn to accurately add noise data after training, thus enabling the image to be converted into the preset style after gradual denoising.

[0100] In summary, the stylized image generation method provided in this disclosure encodes the image to be processed through the encoding layer of an image encoding / decoding model to obtain a first feature; the first feature is used to characterize the image features of the image to be processed. The first feature is fused with noise data through a stylization processing model to obtain a second feature, and prediction noise data is determined based on the second feature. The second feature is then progressively denoised based on the prediction noise data through the stylization processing model to obtain a third feature. The third feature is then decoded through the decoding layer of the image encoding / decoding model to obtain a stylized image corresponding to the image to be processed; the stylized image has the same image content as the image to be processed, and the image content of the stylized image conforms to a preset style. Thus, by converting the image to be processed into the first feature and performing stylization processing in the form of the first feature, the generation efficiency of the stylized image can be improved to a certain extent, since the data volume of the first feature is often smaller than that of the image to be processed. Furthermore, the stylization processing model obtained through training can directly convert the image to be processed into a stylized image conforming to a preset style, thus improving the convenience of obtaining the stylized image to a certain extent. Meanwhile, compared to directly mapping the dimensions of the image to be processed to complete the stylization process, this embodiment of the present disclosure adds noise data to the first feature and then gradually removes noise, which can, to a certain extent, ensure the image detail quality of the generated stylized image.

[0101] Optionally, the embodiments of this disclosure may also include the following steps:

[0102] Step 301: Use the preset sample image as the input to the codec model to be trained, and downsample the preset sample image by the encoding layer in the codec model to be trained at a specified ratio to obtain the hidden state features of the preset sample image in the hidden space.

[0103] Step 302: Decode the hidden state features through the decoding layer in the codec model to be trained to obtain the reconstructed image corresponding to the preset sample image.

[0104] In this step, the preset sample image can be a pre-collected image used to train the codec model. The preset sample image can be the aforementioned sample content image, or it can be other separately collected images; this embodiment does not limit this. The codec model to be trained can be a pre-built model, for example, a self-reconstruction model, which may include an encoding layer and a decoding layer. At the start of training, the model parameters of the encoding and decoding layers in the codec model to be trained can be initialized. During subsequent training, the model parameters of the encoding and decoding layers in the codec model to be trained are continuously optimized, so that the final image codec model can accurately encode the image into a feature representation in a specified space, and simultaneously, accurately decode the feature representation into an image. The step of training the image codec model can be completed before step 101 above. Thus, the image codec model is first trained based on the preset sample image, and then the training data of the stylization model to be trained is encoded into sample features based on the trained image codec model for use by the stylization model to be trained. In this way, the stylization model is jointly trained based on the trained image codec model. This ensures the accuracy of the sample features provided for the stylized model to be trained, thereby ensuring the overall training effect.

[0105] In this context, the latent space refers to a compressed space obtained from the original image space through a certain mapping. The latent state of an image is essentially a compressed representation of the image in a smaller space. Direct inference in the image space would be computationally too expensive, leading to slow processing speeds. The latent space can ignore some redundant information and noise in the original space, focusing on the most critical features, making inference more efficient. This transformation allows the stylization model to be trained to process in a lower-dimensional latent space. The latent state features in the latent space can be understood as a compressed state that retains most of the main information of the original image. Since the encoding / decoding model can approximately reconstruct the image through the decoder, the latent state can also be understood as a state with most redundant information removed.

[0106] Specifically, a convolutional neural network can be used to train an encoder and decoder to achieve a mapping between the image and its hidden states, thus obtaining an image encoding / decoding model. The encoder can downsample the image through convolution to obtain its hidden states, where the downsampling factor is the specified factor, denoted as sf. Assuming the number of hidden state channels is represented by ch, the shape of the hidden state can be represented as (ch, res / sf, res / sf). The larger the downsampling factor sf, the smaller the corresponding hidden state size, and the less image content information is retained. Furthermore, the specified factor can be set according to actual needs; for example, the specified factor can be 4, or it can be 8. This can, to some extent, avoid the problem of the specified factor being too large, resulting in the hidden state size being too small and thus retaining too little image content information.

[0107] Furthermore, the input preset sample image can be encoded by the encoder to obtain latent state features, and the latent state features can be decoded by the decoder to obtain the reconstructed image. For example, the decoder can perform convolution processing on the latent state features based on the convolution kernel parameters to obtain the reconstructed image.

[0108] Step 303: Based on the preset sample image and the reconstructed image, adjust the parameters of the codec model to be trained until the second stopping condition is met, end the training, and use the current codec model to be trained as the image codec model.

[0109] The second stopping condition can be a pre-selected training stopping condition. For example, the second stopping condition could be that the number of training epochs reaches a preset epoch threshold, or that the model loss value of the encoder-decoder model to be trained is less than a preset threshold, etc. Adjusting the parameters of the encoder-decoder model to be trained can specifically involve adjusting the convolutional kernel parameters in the encoder and decoder. For example, the backpropagation algorithm can be used for adjustment.

[0110] In this embodiment, a preset sample image is used as input to the codec model to be trained. The encoding layer of the codec model downsamples the preset sample image at a specified ratio to obtain the latent state features of the preset sample image in the latent space. The decoding layer of the codec model decodes the latent state features to obtain the reconstructed image corresponding to the preset sample image. Based on the preset sample image and the reconstructed image, the parameters of the codec model to be trained are adjusted until a second stopping condition is met, at which point training ends and the current codec model to be trained is used as the image codec model. In this way, the reconstructed image output by the codec model to be trained becomes closer and closer to the real preset sample image as the training process progresses, allowing the codec model to generate features for the image more accurately and to convert features into images more accurately. Since the stylization process requires converting the image into features and restoring the features into an image, the accuracy of the stylized image generated by the subsequent model when actually processing the image can be guaranteed to a certain extent.

[0111] Optionally, the step of adjusting the parameters of the codec model to be trained based on the preset sample image and the reconstructed image may specifically include:

[0112] Step 3031: Determine the absolute pixel error between the preset sample image and the reconstructed image, the content error between the preset sample image and the reconstructed image, and the difference parameters between the reconstructed image and the preset sample image.

[0113] Specifically, the difference between the pixel value of each pixel in the preset sample image and the pixel value of the corresponding pixel in the reconstructed image can be calculated. Then, the absolute value of this difference is taken to obtain the absolute pixel error. For example, if input represents the preset sample image and recon represents the reconstructed image, then the absolute pixel error between the preset sample image and the reconstructed image can be expressed as abs(input-recon).

[0114] Furthermore, the content error between the preset sample image and the reconstructed image can be calculated based on the learned perceptual image patch similarity loss function, namely, the perceptual loss function LPIPS(*). For example, the preset sample image and the reconstructed image can be used as inputs to LPIPS(*), and the output LPIPS(input, recon) can be used as the content error. The content error can be used to measure the difference between the two images.

[0115] Furthermore, the discriminator loss calculated based on the reconstructed image can be used as a difference parameter between the reconstructed image and the preset sample image. The reconstructed image is used as input to the discriminator, and the output of the preset discriminator represents the probability that the input reconstructed image is identified as the preset sample image. The discriminator's role is to distinguish between the preset sample image and the reconstructed image; if the discriminator considers the input to be the preset sample image, it outputs 1; otherwise, it outputs 0. When training the transformation submodule, the generated reconstructed image needs to be identified by the discriminator as the preset sample image, resulting in an output of 1. Furthermore, the difference between the preset discriminator output and 1 can be used as the discriminator loss. The discriminator loss can be represented as: Discriminator(recon).

[0116] Step 3032: Determine the model loss of the codec model to be trained based on the absolute pixel error, the content error, and the difference parameter.

[0117] For example, the model loss of the image encoding and decoding model to be trained can be: loss = abs(input-recon) + lambda_p*LPIPS(input,recon) + lambda_d*Discriminator(recon).

[0118] Here, lambda_p represents the weight of content error, and lambda_d represents the weight of difference parameter. The specific values ​​of lambda_p and lambda_d can be preset. For example, lambda_p and lambda_d can be 0.4 and 0.6 respectively, or they can be 0.3 and 0.7 respectively.

[0119] Step 3033: Adjust the parameters of the codec model to be trained based on the model loss.

[0120] Specifically, the gradient of backpropagation can be calculated based on the calculated model loss. Then, gradient descent is used to adjust the model parameters in the codec model to be trained based on the gradient of backpropagation. Accordingly, if the second stopping condition has not been met at this point, the next round of training can be continued. This cycle is repeated until the second stopping condition is met, at which point the model can be considered converged, and training can be terminated. The codec model to be trained at this point can then be used as the image codec model.

[0121] In this embodiment of the disclosure, the model loss of the codec model to be trained is determined by combining the absolute pixel error between the preset sample image and the reconstructed image, the content error between the preset sample image and the reconstructed image, and the difference parameters between the reconstructed image and the preset sample image. This can ensure the accuracy of the obtained model loss to a certain extent, and thus ensure the adjustment effect when adjusting the parameters of the codec model to be trained based on the model loss.

[0122] It should be noted that, in this embodiment of the disclosure, these images can also be preprocessed before training using preset sample images, sample content images, and sample style images. Specifically, the images can be scaled proportionally so that the shorter side of the image reaches a set size res. Then, the scaled image is cropped at the center to obtain an image of size res*res. Next, the pixels in the image are scaled and translated to normalize the pixel value at each position in the image from [0,255] to [-1,1]. For example, the image can be stored in matrix form, where each position in the matrix is ​​a pixel value in the range [0,255]. The original pixel value of each pixel in the image can be divided by 127.5 and then subtracted by 1 to achieve scaling and translation of the pixels, resulting in the normalized pixel value.

[0123] Because a large range of pixel values ​​can lead to slow learning speed and poor stability of the model, and many activation functions in the model also have an output range of [-1,1]. In this embodiment, normalization to [-1,1] can reduce the range of pixel values ​​and align the range of pixel values ​​with the output range of activation functions, thereby facilitating model parameter learning, accelerating convergence, and improving stability.

[0124] Optionally, the step of generating predicted noise data based on the sample noise features and the sample features corresponding to the sample content map may specifically include:

[0125] Step 1031: Input the sample noise features, the first preset time, and the sample features corresponding to the sample content map into the noise prediction layer of the stylization model to be trained to obtain the prediction noise data corresponding to the first preset time.

[0126] In this embodiment of the disclosure, the first preset time can be a time randomly sampled from a preset time range. For example, the first preset time can be randomly sampled with equal probability from {1, 2, ..., T}. In this embodiment of the disclosure, the first preset time can be considered as the number of steps in which noise is gradually added. {1, 2, ..., T} represents the first step, the second step, ..., the Tth step.

[0127] The noise prediction layer can be a denoising model, such as the UNet model. It can include downsampling and upsampling layers. These layers can include cross-attention layers, which may employ a self-attention mechanism. The upsampling layer can also use skip connections to concatenate features of the same dimension obtained from the downsampling layer, thus avoiding gradient explosion and vanishing problems during training. Specifically, after input to the noise prediction layer, the input data first passes through the downsampling layer, then through the upsampling layer and skip connections to recover predicted Gaussian noise of the same size as the sample features; that is, the predicted noise data.

[0128] Furthermore, in this embodiment of the disclosure, the stylization model to be trained can be a generative model. Exemplarily, the model processing principle of the stylization model to be trained can include two processes: a forward diffusion process and a backward generation process. The forward diffusion process can refer to gradually adding Gaussian noise to the original data until the data becomes random noise, that is, gradually adding noise from x0 to x... T The forward diffusion process can be viewed as a Markov chain, consisting of T steps. Each step of the forward diffusion process is a change in the data x obtained in the previous step. t-1 Add Gaussian noise using the following formula:

[0129]

[0130] in, This represents the variance used in each step, ranging from 0 to 1. A larger t indicates a larger variance is used, satisfying the condition β1 < β2 < ... < β. T If the diffusion step number T is large enough, then the final obtained x T This essentially turns it into random noise. In practical applications, T can be set to 1000. Can be taken The squares of the T-division points are used to ensure the processing effect.

[0131] Furthermore, the reverse generation process refers to the process of gradually denoising and generating new data, that is, from x... T Gradually denoise to obtain x0. Under the guiding condition x c The following denoising formula can be used to gradually reduce noise:

[0132]

[0133] Where, x c This represents the external input condition for guiding the model to denoise. In this embodiment, the sample feature corresponding to the sample content map is x.c . This is the current noise map x t (i.e., the sample noise features corresponding to the first preset time) and condition x c The mean of the Gaussian distribution predicted by the denoising model (i.e., the predicted noise data corresponding to the first preset time). The variance, representing a fixed value, can be calculated directly. Specifically, in,

[0134] One existing approach uses Generative Adversarial Learning (GAN)-based methods to train stylization models. However, due to inherent training instability, this approach struggles to learn complex styles on unstructured datasets, and the generated images often exhibit low detail quality.

[0135] In this embodiment, a diffusion model is used as the stylization model to be trained. Because the diffusion model has better generation capabilities and training stability, it can learn a specified style on unstructured style datasets. Furthermore, this embodiment employs an image encoding / decoding model to encode the image into latent state features. This allows for stylization from the content domain to the style domain based on the diffusion model while reducing processing load, thereby improving the processing speed of the diffusion model, enhancing style learning capabilities, and accelerating the style transfer process. Moreover, the diffusion model is more stable; therefore, it can, to a certain extent, fit a specified style on unstructured datasets, ensuring variations in color and line in the image, thus guaranteeing the detail quality of the generated stylized image.

[0136] Optionally, the step of inputting the sample noise features, the first preset time, and the sample features corresponding to the sample content map into the noise prediction layer of the stylization model to be trained may specifically include:

[0137] Step 1031a: Vectorize the first preset time to obtain time features, and concatenate the sample noise features and the sample features corresponding to the sample content map to obtain concatenated features.

[0138] In this step, the first preset time point can be vectorized, for example, by embedding the first preset time point to obtain its temporal features. The sample noise features and the sample features corresponding to the sample content map can have the same size; for example, both can be represented as (H, W, C), where H represents height, W represents width, and C represents the number of channels. During concatenation, the sample noise features and the sample features corresponding to the sample content map can be concatenated at the channel level. Alternatively, the sample noise features and the sample features corresponding to the sample content map can be directly concatenated together.

[0139] Step 1031b: Input the splicing features and the temporal features into the noise prediction layer of the stylization model to be trained.

[0140] In this step, the concatenated features can first be input into the downsampling layer of the noise prediction layer, and then processed sequentially through each cross-attention layer in the downsampling layer, followed by sequential processing through each cross-attention layer in the upsampling layer. Furthermore, temporal features can be input into each cross-attention layer.

[0141] For example, Figure 3 This is a schematic diagram illustrating a processing procedure according to an exemplary embodiment, such as... Figure 3 As shown, predicted noise represents the predicted noise data. t Describes the noise characteristics of the sample, cond x c This represents the sample features corresponding to the sample content map. `time t` represents the first preset time step. `concat` indicates the concatenation operation, `embed` indicates vectorization processing, `downsample` represents the downsampling layer, `upsample` represents the upsampling layer, and `QKV` represents the cross-attention layer. The dashed arrow pointing from the downsampling layer to the upsampling layer in the diagram indicates the skip connection used in the upsampling layer. After vectorization processing at the first preset time step, the data is injected into the various cross-attention layers in both the downsampling and upsampling layers. The concatenated sample features and sample noise features are then input into the downsampling layer.

[0142] In this embodiment, the temporal features are obtained by vectorizing the first preset time point, and the sample noise features and the sample features corresponding to the sample content map are concatenated to obtain concatenated features. Then, the concatenated features and the temporal features are input into the noise prediction layer of the stylization model to be trained. In this way, the noise prediction layer can easily process the concatenated features.

[0143] Meanwhile, since the forward diffusion process of the stylized model to be trained requires T steps, the denoising process also often requires T steps. In this embodiment of the disclosure, by randomly selecting a first preset time from the T steps, a time feature is added to the input, so that the time feature can be encoded into the network, thereby enabling only one shared denoising model to be trained.

[0144] The steps described above for training the stylization model based on the predicted noise data and the random noise data may specifically include:

[0145] Step 1032: Adjust the parameters of the stylization model to be trained based on the predicted noise data and the random noise data.

[0146] In this embodiment of the disclosure, parameters can be adjusted through backpropagation. Specifically, adjusting the parameters of the stylization model to be trained can involve adjusting all parameters in the denoising model.

[0147] For example, reconstruction loss can be chosen as the loss function, such as L2 loss, with the goal of minimizing the loss function during model training. Specifically, predicted noise data and random noise data can be input into the loss function to calculate the loss value, and then the gradient of backpropagation can be calculated based on the loss value. Afterwards, gradient descent is used to adjust the model parameters in the stylization model to be trained based on the gradient of backpropagation. Correspondingly, if the first stopping condition has not been met, the next round of training can be continued. This cycle continues until the first stopping condition is met, at which point the model can be considered converged, and training can be terminated. The stylization model to be trained at this point is then used as the stylization processing model.

[0148] For example, Figure 4 This is a schematic diagram of a training process according to an exemplary embodiment, combined with Figure 4 As shown in the embodiments of this disclosure, the entire training process of the stylization model to be trained can be specifically as follows: First, randomly sample paired data for training: sample content map (content x) — sample style map (style x). Then, encoding is performed based on the trained image encoding / decoding model: Encoder(x) - Encoder The sample content image (content x) — sample style image (style x) The latent state features encoded into the latent space: cond x c x0 is used for stylistic transformation learning. Then, a first preset time t is obtained by randomly sampling with equal probability from {1, 2, ..., T}. Next, Gaussian noise ∈ with the same shape as x0 is randomly generated, where ∈ ~ N(0, I). x0 is fused with the Gaussian noise ∈ to obtain the noisy hidden state x at time t. t (That is, noisy x in the figure) t ). Among them, noisy x t You can directly use the original data x0 and Gaussian noise ∈ using a preset formula: The calculated x t This refers to the aforementioned sample noise features. Next, x... t time t and xc The input noise prediction layer (i.e., the Conditional Unet Model in the figure) is used to obtain the predicted noise data. Then, based on the predicted noise data and the added Gaussian noise, the model loss is calculated, and the parameters are adjusted. The model loss can be expressed as: If the model has not yet converged, you can repeatedly sample pairs of data used for training to start the next round of training, and repeat the above training process until the model converges.

[0149] In this embodiment, the sample noise features, a first preset time point, and sample features corresponding to the sample content image are first input into the noise prediction layer of the stylization model to be trained, obtaining the predicted noise data corresponding to the first preset time point. Then, the parameters of the stylization model to be trained are adjusted based on the predicted noise data and the added random noise data. This allows the predicted noise data output by the stylization model to increasingly approximate the actual added random noise data during the training process. Since the predicted noise data is required during progressive denoising, the accuracy of the stylized image generated when the model actually processes the image to be processed can be guaranteed to a certain extent.

[0150] Optionally, the step of fusing the first feature with the noise data through the stylization processing model to obtain the second feature, and determining the predicted noise data based on the second feature, may specifically include:

[0151] Step 2021: Fuse the first feature with the random noise data at the second preset time to obtain the second feature.

[0152] The random noise data can be Gaussian noise, and the random noise data at the second preset time can be randomly generated Gaussian noise. Because the stylization processing model is essentially equivalent to gradually adding noise over T steps, the second preset time can belong to {1, 2, ..., T}, and the second preset time can be denoted as S. The random noise data in this step, which is the first feature fusion, can be equivalent to the noise data added over S steps.

[0153] S can be calculated based on the preset diffusion process step parameter ddim_step and diffusion intensity parameter strength. ddim_step can be greater than 0 and not greater than T; for example, ddim_step can be set to 50, 100, or 200. strength can be greater than 0 and less than or equal to 1. For example, strength can be set to be greater than 0.4. For example, strength can be set to 0.5 or 1. S = ddim_step × strength. The method for calculating the second feature is the same as the method for calculating the sample noise feature described above, and will not be repeated here. The second feature can be denoted as:

[0154] The number of steps S for progressive denoising will vary depending on the values ​​of ddim_step and strength. Consequently, the final stylized image will have different effects. For example, the larger ddim_step and strength are, the greater the degree of color and line variation in the stylized image, and the stronger the stylization of the image.

[0155] Step 2022: Input the second feature, the second preset time and the first feature into the noise prediction layer of the stylization processing model to obtain the predicted noise data corresponding to the second preset time.

[0156] The steps described above, which involve progressively denoising the second feature based on the predicted noise data using the stylization processing model to obtain the third feature, may specifically include:

[0157] Step 2031: Perform a denoising operation on the current second feature based on the predicted noise data to obtain the updated second feature.

[0158] Step 2032: If the denoising stop condition is not met, based on the updated second feature, the next time after the second preset time, and the first feature, perform denoising operation on the updated second feature again until the denoising stop condition is met.

[0159] Since the stylization processing model has been trained and converged, its noise prediction layer can accurately generate the predicted noise data corresponding to the second preset time. The denoising stopping condition can be preset, such as reaching time 0, i.e., the number of denoising operations equals S. In this embodiment, starting from t = S, denoising can be performed step-by-step to obtain the second feature at times S-1, S-2, ..., 0. Finally, the second feature at time 0 is the third feature. Of course, the denoising stopping condition can also be other things, such as the cumulative duration of denoising operations reaching a preset duration threshold; this embodiment does not limit this. Thus, by progressively denoising to determine the third feature, the accuracy of the third feature can be ensured to a certain extent, thereby ensuring the efficiency of the stylization conversion.

[0160] Specifically, starting from time t = S, noise is gradually removed, and the second feature of the noisy state is denoted as x. t Time t (i.e., the first preset time) and conditional state x c (That is, the first feature) is input together into the noise prediction layer for S successive denoising operations. The second feature can be concatenated with the first feature, and vectorized at a first preset time point to obtain the temporal feature. The concatenated feature and the temporal feature are then input into the noise prediction layer of the stylization processing model.

[0161] Furthermore, the second feature after performing one denoising operation can be determined based on the following preset formula:

[0162]

[0163] In this case, since the number of denoising steps has been reduced from T to S, therefore... It is equivalent to extracting the original This is part of the process. Specifically, you can first calculate the integer division result of the extraction interval, c = T / / S, in this step. t=1,…,S and UNet(x t ,t,x c ) represents the output predicted noise data. ∈ t σ represents the random noise data added at time t. t It is done according to the preset method: The generated noise intensity parameters. Where σ t The value gradually increases from t=0 to t=S, where ddim_eta represents a pre-set randomness parameter. For example, ddim_eta can be set to 1.

[0164] because x t ,t and UNet(x t ,t,x cSince all ) are definite, x can be obtained step by step starting from t = S. S-1 x S-2 , ..., x1, x0. Among them, x0 obtained in this step is the third feature.

[0165] For example, Figure 5 This is a schematic diagram of another processing flow according to an exemplary embodiment, such as... Figure 5 As shown, the image to be processed (content x) can obtain the first feature cond x after passing through the encoder (i.e., the Encoder in the figure). c By cond x c By fusing with random noise data (i.e., Gaussian noise ∈ in the figure), the second feature noisex can be obtained. s During this process, noise x s That is, the initial noise x t Next, based on the predicted noise data and the current second feature noise x, t The second feature after one denoising operation is calculated, that is, the second feature noise x at the next time t-1. t-1 Then, the third feature, noise x0, is obtained by continuously performing denoising operations. Finally, noise x0 is input into the decoder (i.e., the Decoder in the figure) for decoding to obtain the stylized image styley. In this embodiment of the present disclosure, denoising is performed progressively from the second preset time, which can reduce the number of sampling steps to a certain extent, thereby speeding up the stylization process.

[0166] Figure 6 This is a block diagram illustrating a model generation apparatus according to an exemplary embodiment, such as... Figure 6 As shown, the device 40 may include:

[0167] The first encoding module 401 is configured to perform encoding on sample content maps and sample style maps with the same image content based on the encoding layer in the image encoding and decoding model, to obtain sample features corresponding to the sample content map and the sample style map respectively; the sample content map and the sample style map have different styles, and the sample style map is an image with a preset style.

[0168] The generation module 402 is configured to perform the fusion of sample features corresponding to the sample style map with random noise data based on the style model to be trained, so as to generate sample noise features;

[0169] The first training module 403 is configured to generate predicted noise data based on the sample noise features and the sample features corresponding to the sample content map, and train the stylization model to be trained based on the predicted noise data and the random noise data until a first stopping condition is met, thereby ending the training and using the current stylization model to be trained as the stylization processing model; the stylization processing model is used to convert the image to be processed into a stylized image that conforms to the preset style.

[0170] In one alternative embodiment, the first training module 403 is specifically configured to perform:

[0171] The sample noise features, the first preset time, and the sample features corresponding to the sample content map are input into the noise prediction layer of the stylization model to be trained to obtain the predicted noise data corresponding to the first preset time.

[0172] The parameters of the stylization model to be trained are adjusted based on the predicted noise data and the random noise data.

[0173] In one alternative embodiment, the first training module 403 is further configured to perform:

[0174] The first preset time point is vectorized to obtain time features, and the sample noise features and the sample features corresponding to the sample content map are spliced ​​together to obtain spliced ​​features.

[0175] The splicing features and the temporal features are input into the noise prediction layer of the stylization model to be trained.

[0176] In one alternative embodiment, the device 40 further includes:

[0177] The downsampling module is configured to take a preset sample image as input to the codec model to be trained, and downsample the preset sample image by a specified ratio through the encoding layer in the codec model to be trained, so as to obtain the latent state features of the preset sample image in the latent space.

[0178] The first decoding module is configured to decode the hidden state features through the decoding layer in the codec model to be trained, so as to obtain the reconstructed image corresponding to the preset sample image;

[0179] The adjustment module is configured to perform parameter adjustment on the codec model to be trained based on the preset sample image and the reconstructed image until a second stopping condition is met, thereby ending the training and using the current codec model to be trained as the image codec model.

[0180] In one alternative embodiment, the adjustment module is specifically configured to perform:

[0181] Determine the absolute pixel error between the preset sample image and the reconstructed image, the content error between the preset sample image and the reconstructed image, and the difference parameters between the reconstructed image and the preset sample image;

[0182] The model loss of the codec model to be trained is determined based on the absolute pixel error, the content error, and the difference parameter.

[0183] The parameters of the codec model to be trained are adjusted based on the model loss.

[0184] In summary, the model generation apparatus provided in this embodiment first encodes sample content maps and sample style maps with the same image content based on the encoding layer in the image encoding and decoding model, obtaining sample features corresponding to each sample content map and sample style map. The sample content map and sample style map have different styles, and the sample style map is an image with a preset style. Next, based on the stylization model to be trained, the sample features corresponding to the sample style map are fused with random noise data to generate sample noise features. Finally, prediction noise data is generated based on the sample noise features and the sample features corresponding to the sample content map, and the stylization model to be trained is trained based on the prediction noise data and random noise data until a first stopping condition is met, the training ends, and the current stylization model to be trained is used as the stylization processing model; the stylization processing model is used to convert the image to be processed into a stylized image that conforms to the preset style. In this way, the stylization processing model obtained through training can directly convert the image to be processed into a stylized image that conforms to the preset style, thus improving the convenience of obtaining stylized images to a certain extent. Furthermore, by converting sample images into sample features and training the stylization model in the form of sample features, the efficiency of training the stylization model can be improved to some extent, since the amount of data in the features is often less than the amount of data in the image itself.

[0185] Figure 7 This is a block diagram illustrating a stylized image generation apparatus according to an exemplary embodiment, such as... Figure 7 As shown, the device 50 may include:

[0186] Encoding module 501 is configured to encode the image to be processed through the encoding layer in the image encoding and decoding model to obtain a first feature; the first feature is used to characterize the image features of the image to be processed.

[0187] The determination module 502 is configured to perform a process of fusing the first feature with the noise data through a stylization processing model to obtain a second feature, and to determine the predicted noise data based on the second feature;

[0188] The denoising module 503 is configured to perform stepwise denoising of the second feature based on the predicted noise data through the stylization processing model to obtain the third feature;

[0189] The second decoding module 504 is configured to decode the third feature through the decoding layer in the image encoding and decoding model to obtain a stylized image corresponding to the image to be processed; the stylized image has the same image content as the image to be processed, and the image content of the stylized image conforms to a preset style.

[0190] In one alternative embodiment, the determining module 502 is specifically configured to perform:

[0191] The first feature is fused with random noise data at a second preset time to obtain the second feature;

[0192] The second feature, the second preset time, and the first feature are input into the noise prediction layer of the stylization processing model to obtain the predicted noise data corresponding to the second preset time.

[0193] The noise reduction module 503 is specifically configured to perform:

[0194] Based on the predicted noise data, a denoising operation is performed on the current second feature to obtain the updated second feature;

[0195] If the denoising stop condition is not met, the updated second feature is denoised again based on the updated second feature, the next time after the second preset time, and the first feature, until the denoising stop condition is met.

[0196] In summary, the stylized image generation apparatus provided in this embodiment encodes the image to be processed through the encoding layer in the image encoding / decoding model to obtain a first feature; the first feature is used to characterize the image features of the image to be processed. The first feature is fused with noise data through a stylization processing model to obtain a second feature, and prediction noise data is determined based on the second feature. The second feature is then progressively denoised based on the prediction noise data through the stylization processing model to obtain a third feature. The third feature is decoded through the decoding layer in the image encoding / decoding model to obtain a stylized image corresponding to the image to be processed; the stylized image has the same image content as the image to be processed, and the image content of the stylized image conforms to a preset style. Thus, by converting the image to be processed into the first feature and performing stylization processing in the form of the first feature, the generation efficiency of the stylized image can be improved to a certain extent, since the data volume of the first feature is often smaller than that of the image to be processed. Furthermore, the stylization processing model obtained through training can directly convert the image to be processed into a stylized image conforming to a preset style, thus improving the convenience of obtaining stylized images to a certain extent. Meanwhile, compared to directly mapping the dimensions of the image to be processed to complete the stylization process, this embodiment of the present disclosure adds noise data to the first feature and then gradually removes noise, which can, to a certain extent, ensure the image detail quality of the generated stylized image.

[0197] According to one embodiment of this disclosure, an electronic device is provided, including: a processor and a memory for storing processor-executable instructions, wherein the processor is configured to perform the steps of the method as described in any of the above embodiments when executed.

[0198] According to one embodiment of this disclosure, a storage medium is also provided, which, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the steps of the method as described in any of the above embodiments.

[0199] According to one embodiment of this disclosure, a computer program product is also provided, which includes readable program instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the steps of the method as described in any of the above embodiments.

[0200] Figure 8This is a block diagram illustrating an apparatus for model generation according to an exemplary embodiment. The apparatus 600 may include a processing component 602, a memory 604, a power supply component 606, a multimedia component 608, an audio component 610, an input / output interface 612, a sensor component 614, a communication component 616, and a processor 620. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the stylized image generation method described above. In the exemplary embodiment, a storage medium including instructions is also provided, such as the memory 604 including instructions, which can be executed by the processor 620 of the apparatus 600 to complete the method described above. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0201] Figure 9 This is a block diagram illustrating another apparatus for model generation according to an exemplary embodiment.

[0202] The device 700 may include a processing component 722, a memory 732, an input / output interface 758, a network interface 750, and a power supply component 726. The device 700 may be provided as a server. The application program stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 722 is configured to execute instructions to perform the stylized image generation method described above.

[0203] All user information (including but not limited to user device information, user personal information, etc.) and related data involved in this disclosure are information authorized by the user or by the parties involved.

[0204] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0205] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A model generation method characterized by comprising: The method comprises: Based on the encoding layer in the image coding and decoding model, the sample content graph and the sample style graph with the same image content are encoded respectively, to obtain the sample features corresponding to the sample content graph and the sample style graph respectively; the sample style graph and the sample content graph are different in style, and the sample style graph is a preset style image; Based on the to-be-trained stylization model, the sample feature corresponding to the sample style graph is fused with random noise data to generate sample noise feature; Based on the sample noise feature and the sample feature corresponding to the sample content graph, predicted noise data is generated, and the to-be-trained stylization model is trained based on the predicted noise data and the random noise data until a first stop condition is reached, the training is ended, and the current to-be-trained stylization model is taken as a stylization processing model; the stylization processing model is used to convert a to-be-processed image into a stylized image conforming to the preset style.

2. The method of claim 1, wherein, The method further comprises: The preset sample graph is taken as an input of a to-be-trained coding and decoding model, and a down-sampling of the preset sample graph is performed by an encoding layer in the to-be-trained coding and decoding model according to a specified ratio to obtain a hidden state feature of the preset sample graph in a hidden space; A reconstructed image corresponding to the preset sample graph is obtained by decoding the hidden state feature through a decoding layer in the to-be-trained coding and decoding model; Based on the preset sample graph and the reconstructed image, the to-be-trained coding and decoding model is adjusted in parameters until a second stop condition is reached, the training is ended, and the current to-be-trained coding and decoding model is taken as the image coding and decoding model.

3. The method of claim 2, wherein, The method further comprises: The absolute pixel error between the preset sample graph and the reconstructed image, the content error between the preset sample graph and the reconstructed image, and the difference parameter of the reconstructed image and the preset sample graph are determined; Based on the absolute pixel error, the content error, and the difference parameter, a model loss of the to-be-trained coding and decoding model is determined; 4. The method according to any of claims 1 to 3, characterized in that, The method further comprises: The model loss of the to-be-trained coding and decoding model is adjusted in parameters based on the absolute pixel error, the content error, and the difference parameter. ​ ​ 5. The method of claim 4, wherein, ​ ​ ​ Adjust parameters of the to-be-trained codec model based on the model loss.

6. A stylized image generation method characterized by comprising: The method comprises: encoding a to-be-processed image through an encoding layer in an image codec model to obtain first features; the first features are used to represent image features of the to-be-processed image; fusing the first features and noise data through a stylization processing model to obtain second features, and determining predicted noise data based on the second features; gradually denoising the second features according to the predicted noise data through the stylization processing model to obtain third features; decoding the third features through a decoding layer in the image codec model to obtain a stylized image corresponding to the to-be-processed image; the stylized image has the same image content as the to-be-processed image, and the image content of the stylized image conforms to a preset style.

7. The method of claim 6, wherein, The method comprises: fusing the first features and random noise data at a second preset time to obtain second features; inputting the second features, the second preset time and the first features into a noise prediction layer of the stylization processing model to obtain predicted noise data corresponding to the second preset time; The method comprises: performing a denoising operation on the current second features based on the predicted noise data to obtain updated second features; if a denoising stop condition is not reached, performing a denoising operation again on the updated second features based on the updated second features, a next time of the second preset time and the first features, until the denoising stop condition is reached.

8. A model generation apparatus characterized by comprising: The device comprises: a first encoding module configured to encode a sample content image and a sample style image respectively based on an encoding layer in an image codec model, the sample content image and the sample style image having the same image content; the sample content image and the sample style image have different styles, and the sample style image is a preset style image; a generation module configured to fuse the sample feature corresponding to the sample style image and random noise data based on a to-be-trained stylization model to generate a sample noise feature; a first training module configured to generate predicted noise data based on the sample noise feature and the sample feature corresponding to the sample content image, and train the to-be-trained stylization model based on the predicted noise data and the random noise data until a first stop condition is reached, end the training and use the current to-be-trained stylization model as a stylization processing model; the stylization processing model is used to convert a to-be-processed image into a stylized image conforming to the preset style.

9. An electronic device, comprising: comprises: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method of any one of claims 1 to 7.

10. A storage medium, characterized by When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device performs the method as claimed in any one of claims 1 to 7.

Images