Model training, image stylization method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a model training method, an image stylization method, a device, equipment and a medium. The method comprises: obtaining a first training sample set, each group of training samples comprising an original image and a stylized image of a target object, performing the same motion simulation on the target object in the original image and the stylized image to obtain a motion simulated original image and a motion simulated stylized image, generating an optical flow image according to the original image and the corresponding motion simulated original image, and training a first stylized image generation model according to the training samples, the optical flow image and the motion simulated stylized image, so that adjacent frames in a video are created by motion simulation, and the optical flow image is introduced, so that the motion change between the stylized images of adjacent frames can be consistent with the motion change between adjacent frames when the model generates the stylized images, avoiding unpredictable flicker between inter-frame stylized images, thereby improving the stability of the stylized video.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a model training method, an image stylization method, a model training device, an image stylization device, an electronic device and a storage medium. BACKGROUND

[0002] With the development of computer technology, the use of intelligent terminals has been widely popularized, and more and more application programs have been developed to facilitate and enrich people's work and life. At present, many application programs are committed to providing more personalized and better visual effects for intelligent terminal users, such as filter effects, sticker effects, deformation effects, etc.

[0003] Among them, the effect of changing the style of the image is a common visual effect, which can change the image into another style by changing the color, texture and other attributes of the image.

[0004] The existing image stylization generation model usually only considers the generation problem from image to image, without considering the time sequence relationship between frames in the video, and only stylizes each frame separately, so the shape contour in the stylized image of adjacent frames may differ greatly. Therefore, the existing image stylization generation model performs very poorly in video shooting, specifically, there is unpredictable flicker problem between frames in the shot video. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a model training method, an image stylization method, a model training device, an image stylization device, an electronic device and a storage medium, so as to solve the problem of unpredictable flicker between frames in the shot video.

[0006] In order to solve the above problems, in the first aspect of the present application, a model training method is first provided, comprising:

[0007] obtaining a first training sample set, each group of training samples in the first training sample set comprising an original image containing a target object and a corresponding stylized image;

[0008] For each group of training samples, the target object in the original image and the target object in the corresponding stylized image are subjected to the same motion simulation, to obtain a motion simulated original image and a corresponding motion simulated stylized image;

[0009] According to the original image and the corresponding motion simulated original image, a corresponding optical flow image is generated;

[0010] According to the training sample, the optical flow image, and the stylized image after motion simulation, a first stylized image generation model is trained.

[0011] Optionally, the motion simulation includes at least one of rotation, translation, random deformation, and motion blur.

[0012] Optionally, the training of the first stylized image generation model according to the training sample, the optical flow image, and the stylized image after motion simulation includes:

[0013] The original image, the stylized image, and the optical flow image are spliced in the channel dimension to obtain an input image;

[0014] According to the input image and the stylized image after motion simulation, a second training sample set is formed, each training sample in the second training sample set including one input image and a corresponding stylized image after motion simulation;

[0015] According to the second training sample set, the first stylized image generation model is trained.

[0016] Optionally, the training of the first stylized image generation model according to the second training sample set includes:

[0017] According to a first proportion of the image loss and the adversarial loss, the image loss and the adversarial loss are weighted and summed to obtain a first total loss, the difference between the first proportion and 1 being less than a first preset value;

[0018] Based on the first total loss, the first stylized image generation model is trained according to the second training sample set until the first total loss converges.

[0019] After the first total loss converges, according to a second proportion of the image loss and the adversarial loss, the image loss and the adversarial loss are weighted and summed to obtain a second total loss, the second proportion being greater than or equal to a second preset value;

[0020] Based on the second total loss, the first stylized image generation model is trained according to the second training sample set until the second total loss converges.

[0021] In a second aspect of the embodiment of the present application, an image stylization method is also provided, including:

[0022] Obtaining video data;

[0023] input a previous frame image of the video data into a second stylized image generation model for stylized processing to obtain a corresponding previous frame stylized image, the second stylized image generation model being trained according to a first training sample set, each group of training samples in the first training sample set comprising an original image containing a target object and a corresponding stylized image;

[0024] generate a corresponding optical flow image according to the current frame image and the previous frame image of the video data;

[0025] input the previous frame image, the previous frame stylized image and the optical flow image into a first stylized image generation model for stylized processing to obtain a corresponding current frame stylized image, the first stylized image generation model being obtained by acquiring the first training sample set, for each group of training samples, performing the same motion simulation on the target object in the original image and the target object in the corresponding stylized image to obtain a motion simulated original image and a corresponding motion simulated stylized image, generating a corresponding optical flow image according to the original image and the corresponding motion simulated original image, and training the first stylized image generation model according to the training sample, the optical flow image and the motion simulated stylized image.

[0026] Optionally, the method further comprises:

[0027] generate a third training sample set according to the current frame image and the corresponding current frame stylized image, each group of training samples in the third training sample set comprising a current frame image and a corresponding current frame stylized image;

[0028] train a third stylized image generation model according to the third training sample set;

[0029] acquire target video data;

[0030] input a frame image in the target video data into the third stylized image generation model for stylized processing to obtain a corresponding stylized image.

[0031] In a third aspect of the embodiment of the present application, a model training device is further provided, comprising:

[0032] a sample set acquisition module configured to acquire a first training sample set, each group of training samples in the first training sample set comprising an original image containing a target object and a corresponding stylized image;

[0033] a motion model module configured to, for each group of training samples, perform the same motion simulation on the target object in the original image and the target object in the corresponding stylized image to obtain a motion simulated original image and a corresponding motion simulated stylized image;

[0034] an image generation module configured to generate a corresponding optical flow image according to the original image and the original image after motion simulation;

[0035] a model training module configured to train a first stylized image generation model according to the training sample, the optical flow image and the stylized image after motion simulation.

[0036] Optionally, the motion simulation comprises at least one of rotation, translation, random deformation and motion blur.

[0037] Optionally, the model training module comprises:

[0038] an image splicing sub-module configured to perform image splicing on the original image, the stylized image and the optical flow image in a channel dimension to obtain an input image;

[0039] a sample set composition sub-module configured to compose a second training sample set according to the input image and the stylized image after motion simulation, each training sample in the second training sample set comprising one input image and one corresponding stylized image after motion simulation;

[0040] a model training sub-module configured to train the first stylized image generation model according to the second training sample set.

[0041] Optionally, the model training sub-module comprises:

[0042] a first summing unit configured to perform weighted summation on the image loss and the adversarial loss according to a first proportion of the image loss and the adversarial loss to obtain a first total loss, the first proportion being less than a first preset value by 1;

[0043] a first training unit configured to train the first stylized image generation model according to the second training sample set based on the first total loss until the first total loss converges;

[0044] a second summing unit configured to perform weighted summation on the image loss and the adversarial loss according to a second proportion of the image loss and the adversarial loss to obtain a second total loss after the first total loss converges, the second proportion being greater than or equal to a second preset value;

[0045] a second training unit configured to train the first stylized image generation model according to the second training sample set based on the second total loss until the second total loss converges.

[0046] In a fourth aspect, the present application provides an image stylization device, comprising:

[0047] The data acquisition module is configured to acquire video data.

[0048] The first stylization processing module is configured to input a previous frame image of the video data into a second stylization image generation model for stylization processing to obtain a corresponding previous frame stylization image.

[0049] The image generation module is configured to generate a corresponding optical flow image according to a current frame image and the previous frame image of the video data.

[0050] The second stylization processing module is configured to input the previous frame image, the previous frame stylization image and the optical flow image into a first stylization image generation model for stylization processing to obtain a corresponding current frame stylization image.

[0051] Optionally, the device further comprises:

[0052] The sample set composition module is configured to compose a third training sample set according to the current frame image and the corresponding current frame stylization image.

[0053] The model training module is configured to train a third stylization image generation model according to the third training sample set.

[0054] The target video acquisition module is configured to acquire target video data.

[0055] The third stylization processing module is configured to input a frame image in the target video data into the third stylization image generation model for stylization processing to obtain a corresponding stylization image.

[0056] In another aspect of the embodiment of the present application, an electronic device is also provided, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus.

[0057] The memory is configured to store a computer program.

[0058] a processor configured to implement a method recited by any of the examples above when executing instructions stored on a memory.

[0059] In yet another aspect of the embodiments, a computer readable storage medium is provided, having instructions stored therein, which when executed on a computer, cause the computer to perform any of the methods described above.

[0060] In yet another aspect of the embodiments, a computer program product is provided, having instructions stored therein, which when executed on a computer, cause the computer to perform any of the methods described above.

[0061] The embodiments of the present application provide a model training method and device, electronic equipment and storage medium, by obtaining a first training sample set, each group of training samples in the first training sample set includes an original image containing a target object and a corresponding stylized image, for each group of training samples, the target object in the original image and the target object in the corresponding stylized image are subjected to the same motion simulation, to obtain a motion simulated original image and a corresponding motion simulated stylized image, according to the original image and the corresponding motion simulated original image, a corresponding optical flow image is generated, and according to the training sample, the optical flow image and the motion simulated stylized image, a first stylized image generation model is trained, so that the adjacent frames in the video are manufactured by motion simulation for model training, and the optical flow image between the original image and the motion simulated original image is introduced as the input of the model, so that the first stylized image generation model can utilize the correlation between adjacent frames when generating stylized images, the motion changes between the generated stylized images of adjacent frames can be consistent with the motion changes between adjacent frames, the unpredictable flicker existing between the inter-frame stylized images is avoided, and then the stability of the stylized video is improved. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced.

[0063] Figure 1 A step flowchart of an embodiment of a model training method of the present application is shown;

[0064] Figure 2 A step flowchart of an embodiment of a model training method of the present application is shown;

[0065] Figure 3 A step flowchart of an embodiment of a model training method of the present application is shown;

[0066] Figure 4A step flow chart of an embodiment of an image stylization method of the present application is shown.

[0067] Figure 5 A structural block diagram of an embodiment of a model training device of the present application is shown.

[0068] Figure 6 A structural block diagram of an embodiment of an image stylization device of the present application is shown.

[0069] Figure 7 A schematic diagram of an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0070] To make the above objectives, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the drawings and specific embodiments.

[0071] Reference Figure 1 A step flow chart of an embodiment of a model training method of the present application is shown, which can specifically include the following steps:

[0072] Step 101, a first training sample set is obtained, each group of training samples in the first training sample set including an original image containing a target object and a corresponding stylized image.

[0073] In an embodiment of the present application, the target object can be pre-set according to actual needs, which is not limited herein. For example, the target object can be at least one of a person, an animal or an object, or can be at least one of one or more parts of a person, an animal or an object.

[0074] In an embodiment of the present application, stylization is to change the color, texture and other attributes of an image, so as to change the image into another style, for example, a black-and-white line style, a color painting style, a cartoon style, a hand-drawn style, a 3D (3-dimension) style, etc., or other arbitrary applicable styles, which are not limited in the present embodiment.

[0075] For example, face stylization is to convert a face image into a specific stylized face image, such as a sketch portrait style, a cartoon image (animation) style, a painting style, etc.

[0076] In an embodiment of the present application, the first training sample set includes pairs of original images and stylized images, each pair of corresponding original image and stylized image forming a group of training samples. Wherein, each pair of corresponding original image and stylized image has the same image content. That is, the stylized image in each group of training samples can be obtained by stylizing the original image in the group of training samples.

[0077] In an optional embodiment of the present application, a specific generation manner of the first training sample set comprises: performing first target object detection on video frames in the first video data to obtain a plurality of video frames containing the first target object, taking the plurality of video frames containing the first target object as training samples, training a stylized image pair generation model, and then generating the first training sample set by using the stylized image pair generation model, wherein each group of training samples in the first training sample set comprises an original image containing a second target object and a corresponding stylized image, and the second target object is associated with the first target object in shape.

[0078] The stylized image pair generation model is a machine learning model that can be used to generate image pairs, i.e., original images and corresponding stylized images, after being trained. The trained first stylized image generation model can output original images and corresponding stylized images as needed. The stylized image pair generation model needs to be trained using images with stylization, and the obtained model is referred to as a stylized image pair generation model.

[0079] For example, the stylized image pair generation model can use the network architecture of StyleGAN (StyleGAN). The StyleGAN can be constructed based on various types of generative adversarial networks (GAN).

[0080] Taking a cartoon episode as an example, the first target object is a face, and the first video data is the video data of the cartoon episode. For the video data of a cartoon episode, a multimedia video processing tool (such as FFmpeg) is used to frame, for example, 3 frames are extracted per second. A face detection model specialized in cartoon is used to detect faces and facial feature points in the video frames to obtain a plurality of video frames containing cartoon faces. The plurality of video frames containing cartoon faces are used as training samples to train the stylized image pair generation model.

[0081] The stylized image pair generation model is used to generate original images containing real faces and corresponding stylized images. The real faces are associated with the cartoon faces in shape. The training sample set composed of the original images containing real faces and the corresponding stylized images is referred to as the first training sample set.

[0082] In an optional embodiment of the present application, the second target object in the original image in the first training sample set generated according to the above manner is used to perform affine transformation on the stylized image in the first training sample set, so that the difference between the shape of the second target object in the stylized image and the shape of the second target object in the original image is reduced.

[0083] Affine transformation refers to a linear transformation of a vector space and a translation in geometry, which transforms one vector space into another vector space, i.e., from one two-dimensional coordinate system to another two-dimensional coordinate system, and belongs to linear transformation. For example, the warp algorithm is an algorithm for implementing affine transformation.

[0084] Since there is a difference between the shape of the face in the stylized image generated by the stylized image pair generation model and the shape of the face in the original image. The warp algorithm is used to deform the cartoon face in the stylized image according to the face shape of the real face in the original image, so that the difference between the shape of the face in the stylized image and the shape of the face in the original image is reduced, thereby reducing the training difficulty of the stylized image generation model.

[0085] In an optional embodiment of the present application, since the shapes and angles of the first target object in the plurality of video frames obtained from the first video data are different, the training of the stylized image pair generation model is difficult. Therefore, before training the stylized image pair generation model by taking the plurality of video frames containing the first target object as training samples, the method can further include: performing affine transformation on the plurality of video frames containing the first target object according to the reference first target object contained in the reference image to obtain a plurality of video frames in which the first target object is aligned with the reference first target object.

[0086] The reference first target object contained in the reference image refers to a reference first target object contained in an image whose shape and angle meet the reference requirements. The plurality of video frames containing cartoon faces are subjected to affine transformation to deform according to the face shape of the reference face in the reference image, and the plurality of video frames in which the cartoon faces are aligned with the reference faces are obtained as training samples.

[0087] Step 102, for each group of training samples, the same motion simulation is performed on the target object in the original image and the target object in the corresponding stylized image to obtain a motion simulated original image and a corresponding motion simulated stylized image.

[0088] In the embodiment of the present application, motion simulation is a change to the target object in the image to simulate various motions of the target object. For example, motion simulation can include rotation, translation, random deformation, blur caused by motion, or other arbitrary changes, and the present application does not limit the changes.

[0089] In the embodiment of the present application, for each group of training samples in the first training sample set, the same motion simulation is performed on the target object in the original image and the target object in the corresponding stylized image to obtain a motion simulated original image and a corresponding motion simulated stylized image.

[0090] In the embodiments of the present application, a plurality of motion simulations can be performed on a set of training samples to obtain a plurality of pairs of motion-simulated original images and motion-simulated stylized images. The specific type of motion simulation performed on the training samples is not limited.

[0091] In an optional embodiment of the present application, the motion simulation includes at least one of rotation, translation, random deformation, and motion blur. The rotation is a rotation processing of a target object by a target angle. The translation is a movement processing of the target object by a target distance in a target direction. The random deformation is a random deformation processing of the target object. The motion blur is a blur processing of the target object caused by motion.

[0092] In step 103, a corresponding optical flow image is generated according to the original image and the corresponding motion-simulated original image.

[0093] In the embodiments of the present application, the optical flow is the instantaneous velocity of the pixel motion of a spatial moving object on an observation imaging plane, and is a method of finding the correspondence between the previous frame and the current frame by using the change of the pixel in the time domain and the correlation between adjacent frames, so as to calculate the motion information of the object between adjacent frames. Since the pixel coordinates of the corresponding pixels in the original image and the corresponding motion-simulated original image are different in the respective images, the displacement relationship of the corresponding pixels in the original image and the corresponding motion-simulated original image is found by registration, and the optical flow image is obtained. The optical flow image can be calculated by an optical flow algorithm on the registered image pair. Common optical flow algorithms mainly include the horn-schunck algorithm, the Lucas-Kanade algorithm, and the like, and can be selected as needed.

[0094] In the embodiments of the present application, when the original image and the corresponding motion-simulated original image are registered, the pixels of the original image are first mapped into a first coordinate system (for example, a wide-angle coordinate system) of the motion-simulated original image by coordinate mapping, so that the pixels of the motion-simulated original image and the pixels of the original image are in the same coordinate system (i.e., the first coordinate system).

[0095] In step 104, a first stylized image generation model is trained according to the training samples, the optical flow image, and the motion-simulated stylized image.

[0096] In the embodiment of the present application, the first stylized image generation model is a machine learning model that can be used to stylize an original image after being trained. After providing an image, the trained first stylized image generation model can output a corresponding stylized image. The first stylized image generation model needs to be trained using paired images, and the obtained model is referred to as the first stylized image generation model.

[0097] For example, the first stylized image generation model can use the network architecture of a generative adversarial network. The generative adversarial network can be constructed based on various types of generative adversarial networks (GAN). The main structure of GAN includes a generator G and a discriminator D. The generator G is used to stylize the original image in the training sample and output a generated image. The discriminator D is used to distinguish the stylized image and the authenticity of the generated image in the training sample, that is, to determine whether the stylized image is real or fake, and whether the generated image is real or fake.

[0098] In the embodiment of the present application, considering that there may be motion between the current frame and the previous frame in the video, only stylizing each frame separately, the shape contours in the stylized images of adjacent frames may differ greatly, causing the video composed of continuous multiple frames of stylized images to have unpredictable flicker problems. In order to eliminate this unpredictable flicker problem, when generating the stylized image of the current frame, the first stylized image generation model proposes to use the original image of the previous frame, the stylized image of the previous frame, and the optical flow image representing the motion information between the previous frame and the current frame as the basis for generation. Considering that the previous frame is equivalent to the original image and the corresponding stylized image, and the current frame is equivalent to the original image after motion simulation and the corresponding stylized image, in the training process, the first stylized image generation model needs to use the original image and the corresponding stylized image in the training sample, the optical flow image and the stylized image after motion simulation for training, so that the first stylized image generation model is not only stylized for each frame separately, but also considers the adjacent images and the motion information between adjacent images for stylization.

[0099] In one implementation, the original image and the corresponding stylized image in the training sample, and the optical flow image are converted into one input image, and the stylized image after motion simulation is converted into another input image. The two input images form the paired images of the above-mentioned input model. The manner of converting the original image and the corresponding stylized image in the training sample, and the optical flow image into one input image is not limited.

[0100] According to the embodiment of the present application, by acquiring a first training sample set, each group of training samples in the first training sample set includes an original image containing a target object and a corresponding stylized image, for each group of training samples, the target object in the original image and the target object in the corresponding stylized image are subjected to the same motion simulation to obtain a motion simulated original image and a corresponding motion simulated stylized image, the corresponding optical flow image is generated according to the original image and the corresponding motion simulated original image, and the first stylized image generation model is trained according to the training sample, the optical flow image and the motion simulated stylized image, so that the adjacent frames in the video are manufactured by motion simulation for model training, and the optical flow image between the original image and the motion simulated original image is introduced as the input of the model, so that the first stylized image generation model can utilize the correlation between adjacent frames when generating stylized images, the motion change between the generated stylized images of adjacent frames can be consistent with the motion change between adjacent frames, the unpredictable flicker existing between the inter-frame stylized images is avoided, and then the stability of the stylized video is improved.

[0101] In an optional embodiment of the present application, as shown in Figure 2 The step 104 includes:

[0102] Step 1041, image splicing is performed on the original image, the stylized image and the optical flow image in the channel dimension to obtain an input image.

[0103] Step 1042, a second training sample set is composed according to the input image and the motion simulated stylized image, each group of training samples in the second training sample set includes an input image and a corresponding motion simulated stylized image.

[0104] Step 1043, the first stylized image generation model is trained according to the second training sample set.

[0105] Image splicing is a method of splicing multiple images into a larger image, and the output of image splicing is the union of multiple input images. The original image, the stylized image and the optical flow image are spliced in the channel dimension to obtain an image, which is recorded as an input image. The above image splicing method is adopted for each group of training samples and the corresponding optical flow image to obtain the corresponding input image. When image splicing, the splicing order of the original image, the stylized image and the optical flow image needs to be fixed, that is, the same splicing order is adopted when the model is trained and the stylized image is generated by using the model, and the specific splicing order is not limited in the embodiment of the present application.

[0106] The second training sample set comprises pairs of input images and stylized images after motion simulation, and each pair of corresponding input images and stylized images after motion simulation constitutes a group of training samples. The first stylized image generation model is trained by using multiple groups of training samples in the second training sample set, so as to obtain the trained first stylized image generation model.

[0107] For example, the training sample of the first training sample set is an input-output pair image <x, y>, wherein x represents an original image, and y represents a stylized image. The same motion simulation is performed on each group <x, y> to obtain a pair of images <x1, y1> after motion simulation. According to x and corresponding x1, a flow image x2 is generated. Then, x, y and x2 are spliced in the channel dimension, and the spliced matrix is taken as an input, denoted as X0, y1 is taken as an output, and the training sample of the second training sample set is a pair of images <X0, y1>.

[0108] In an optional embodiment of the present application, as shown in Figure 3 The step 1043 comprises:

[0109] In step 10431, the image loss and the adversarial loss are weighted and summed according to a first proportion of the image loss and the adversarial loss, to obtain a first total loss, and a difference between the first proportion and 1 is less than a first preset value.

[0110] In step 10432, based on the first total loss, the first stylized image generation model is trained according to the second training sample set until the first total loss converges.

[0111] In step 10433, after the first total loss converges, the image loss and the adversarial loss are weighted and summed according to a second proportion of the image loss and the adversarial loss, to obtain a second total loss, and the second proportion is greater than or equal to a second preset value.

[0112] In step 10434, based on the second total loss, the first stylized image generation model is trained according to the second training sample set until the second total loss converges.

[0113] In the embodiment of the present application, the first stylized image generation model adopts the network architecture of the adversarial generation network.

[0114] In the training process, there may be differences between the input image and the corresponding generated image in each group of training samples. The same pixels in the corresponding input image and generated image can be compared one by one to determine the difference value of each pixel, and then the image loss between the input image and the generated image is determined according to the difference value of each pixel. For example, L1 Loss (absolute error distance) takes absolute error as distance.

[0115] The adversarial loss can include a true sample loss corresponding to the stylized image after motion simulation, a fake sample true loss corresponding to the generated image, and a fake sample false loss corresponding to the generated image.

[0116] Wherein, since the discriminative network needs to judge multiple stylized images after motion simulation as true samples (i.e. real samples, with a probability of 1), but in the actual training process, the probability of each stylized image after motion simulation being judged as true by the discriminative network may not be 1, at this time, an adversarial loss can be determined based on the judgment of the true or false probability of the stylized image after motion simulation, which is defined as the true sample loss corresponding to the stylized image after motion simulation in the embodiments of the present disclosure.

[0117] Since the discriminative network needs to judge multiple generated images as fake samples (i.e. generated samples, with a probability of 0), but in the actual training process, the probability of each generated image being judged as true by the discriminative network may not be 0, at this time, another adversarial loss can be determined based on the judgment of the true or false probability of the generated image, which is defined as the fake sample true loss corresponding to the generated image in the embodiments of the present disclosure.

[0118] Since the generative network needs to reduce the gap between the generated sample (generated image) and the real sample (stylized image after motion simulation) as much as possible, i.e. the generative network tries to make the discriminative network make a mistake and judge multiple generated images as true samples. At this time, another adversarial loss can be determined based on the judgment of the true or false probability of the generated image caused by the generative network (mistaken judgment), which is defined as the fake sample false loss corresponding to the generated image in the embodiments of the present disclosure.

[0119] Considering that the image loss and the adversarial loss corresponding to each set of training samples contribute differently to the optimization of the network, in the embodiments of the present disclosure, the ratio between the image loss and the adversarial loss is set to adjust the importance of each loss.

[0120] The process of training the first stylized image generation model according to the second training sample set can be divided into two stages.

[0121] In the first stage, the ratio between the image loss and the adversarial loss is set to a first ratio, and the difference between the first ratio and 1 is less than a first preset value. Wherein, the first preset value is not limited, and is set according to actual needs, so that the first ratio is close to 1, that is, the importance of the image loss and the adversarial loss is close, for example, the first ratio is set to 1. According to the first ratio of the image loss and the adversarial loss, the image loss and the adversarial loss are weighted and summed to obtain a total loss, denoted as a first total loss.

[0122] Then in the training process of the first stage, the model parameters of the first stylized image generation model are adjusted according to the first total loss corresponding to each set of training samples in the second training sample set, and the first stylized image generation model is optimized. After adjustment for multiple sets of training samples, the first total loss converges, and the first stage of training of the first stylized image generation model is completed.

[0123] In the second stage, the ratio between the image loss and the adversarial loss is set to a second ratio, and the second ratio is greater than a second preset value. The second preset value is not limited, and is set according to actual needs, so that the second ratio is larger, that is, the importance of the image loss is higher than that of the adversarial loss. For example, the second ratio is set to 200. The image loss and the adversarial loss are weighted and summed according to the second ratio of the image loss and the adversarial loss, and the total loss obtained is denoted as a second total loss.

[0124] Then in the training process of the second stage, the model parameters of the first stylized image generation model are adjusted according to the second total loss corresponding to each set of training samples in the second training sample set, and the first stylized image generation model is optimized. After adjustment for multiple sets of training samples, the second total loss converges, and the second stage of training of the first stylized image generation model is completed.

[0125] The inventors of the present disclosure found through a large number of experiments that the first ratio of the image loss and the adversarial loss is first set to be close to 1, the importance of the image loss and the adversarial loss is close, the first stylized image generation model is optimized according to the first total loss, which is beneficial to the stylization effect of the first stylized image generation model. The second ratio of the image loss and the adversarial loss is then set to be larger, the image loss has a greater importance than the adversarial loss, the first stylized image generation model is optimized according to the second total loss, which is beneficial to the output of the first stylized image generation model being closer to the input, further eliminating the problem of unstable generated images caused by stylization, and then avoiding unpredictable flicker of inter-frame stylized images, thereby improving the stability of the stylized video.

[0126] Referring to Figure 4 , a step flowchart of an embodiment of an image stylization method of the present application is shown, which can specifically include the following steps:

[0127] Step 201, obtaining video data.

[0128] In the embodiment of the present application, the video data can be video data obtained by real-time shooting of an electronic device, or video data obtained from a video that has been shot. For example, the video data can be video data of a video stored locally by an electronic device, a video sent by another electronic device, or a video on the Internet.

[0129] Step 202, input the previous frame image of the video data into a second stylized image generation model for stylization processing to obtain a corresponding previous frame stylized image, the second stylized image generation model being trained according to a first training sample set, each group of training samples in the first training sample set comprising an original image containing a target object and a corresponding stylized image.

[0130] In the embodiment of the present application, the second stylized image generation model is a machine learning model that can be used to stylize an original image after training. After providing an image, the trained second stylized image generation model can output a corresponding stylized image. The second stylized image generation model is trained using the first training sample set, and the obtained model is referred to as the second stylized image generation model. For example, the second stylized image generation model can use the network architecture of the generative adversarial network.

[0131] In the embodiment of the present application, the video data is composed of a plurality of frame images. When stylizing the video data, the video data can be processed frame by frame, or every several frame images can be processed, and the embodiment of the present application does not limit this. The current frame image and the previous frame image can be adjacent frames or non-adjacent frames. For example, the current frame image and the previous frame image are two frame images separated by one frame.

[0132] In the embodiment of the present application, the previous frame image of the video data is input into the second stylized image generation model for stylization processing to generate a stylized image, which is referred to as the corresponding previous frame stylized image.

[0133] Step 203, generating a corresponding optical flow image according to the current frame image and the previous frame image of the video data.

[0134] In the embodiment of the present application, the displacement relationship of the pixels at the corresponding positions in the previous frame image and the current frame image is found by registration to obtain the corresponding optical flow image.

[0135] Step 204, inputting the previous frame image, the previous frame stylized image, and the optical flow image into a first stylized image generation model for stylization processing to obtain a corresponding current frame stylized image, the first stylized image generation model being obtained by acquiring the first training sample set, for each group of training samples, performing the same motion simulation on the target object in the original image and the target object in the corresponding stylized image to obtain a motion simulated original image and a corresponding motion simulated stylized image, generating a corresponding optical flow image according to the original image and the corresponding motion simulated original image, and training the first stylized image generation model according to the training sample, the optical flow image, and the motion simulated stylized image.

[0136] In the embodiment of the present application, the last frame image, the last frame stylized image obtained in step 202, and the corresponding optical flow image generated in step 203 are input into the first trained stylized image generation model. The stylized processing is performed by the first stylized image generation model, and a generated stylized image is recorded as a corresponding current frame stylized image.

[0137] In the embodiment of the present application, the last frame image, the last frame stylized image and the corresponding optical flow image are spliced in the channel dimension to obtain an image input into the first stylized image generation model.

[0138] For example, for video data, it is first decoded into a picture sequence, the first frame image Xin1 is used to generate the first frame stylized image Yout1 by the second stylized image generation model, the corresponding optical flow image Xflow is generated according to the second frame image Xin2 and Xin1, and Xin1, Yout1 and Xflow are spliced in the channel dimension to obtain an input of the first trained stylized image generation model. The first stylized image generation model outputs the second frame stylized image Yout2, and the stylized images of all frames of the video are obtained in turn.

[0139] In the embodiment of the present application, the above-mentioned method can be used for stylizing the frame images other than the first frame image in the video data.

[0140] In an optional embodiment of the present application, a third training sample set is further composed according to the current frame image and the corresponding current frame stylized image, each group of training samples in the third training sample set includes a current frame image and a corresponding current frame stylized image, a third stylized image generation model is trained according to the third training sample set, target video data is obtained, and the frame images in the target video data are input into the third stylized image generation model for stylized processing to obtain corresponding stylized images.

[0141] Since the stylized processing process of the first stylized image generation model is relatively complex, it affects the real-time shooting speed of electronic devices with limited computing power such as mobile terminals. Therefore, in the embodiment of the present application, the first stylized image generation model is used to compose a third training sample set according to the current frame image and the corresponding current frame stylized image. A large amount of video data is stylized to obtain the third training sample set, wherein each group of training samples includes a current frame image and a current frame stylized image. The third training sample set can be used as stable video training data.

[0142] The third stylized image generation model is a machine learning model that, after being trained, can be used to stylize an original image. After providing an image, the trained third stylized image generation model can output a corresponding stylized image. The third stylized image generation model is trained using the third training sample set, and the obtained model is denoted as the third stylized image generation model. For example, the third stylized image generation model can adopt the network architecture of a generative adversarial network. The third stylized image generation model can be a lightweight model.

[0143] The target video data can be video data obtained by real-time shooting of an electronic device, or can be video data obtained from a video that has been shot. For example, the target video data can be video data of a video stored locally by an electronic device, a video sent by another electronic device, or a video on the Internet, etc.

[0144] The target video data is composed of multiple frame images. When stylizing the target video data, the target video data can be processed frame by frame, or every several frames can be processed. The embodiments of the present application do not limit this.

[0145] The frame images of the target video data are input into the trained third stylized image generation model for stylization processing to generate corresponding stylized images.

[0146] Since the third training sample set is a stable video training data, the third stylized image generation model can also avoid unpredictable flickering of the stylized images between frames, thereby improving the stability of the stylized video. In addition, the stylization process of the third stylized image generation model can be simpler than that of the first stylized image generation model, and therefore the third stylized image generation model can meet the real-time shooting requirements of electronic devices with limited computing power such as mobile terminals.

[0147] According to the embodiment of the present application, by acquiring video data, inputting a previous frame image of the video data into a second stylized image generation model for stylized processing, a corresponding previous frame stylized image is obtained, the second stylized image generation model is trained according to a first training sample set, each group of training samples in the first training sample set includes an original image containing a target object and a corresponding stylized image, according to a current frame image and a previous frame image of the video data, a corresponding optical flow image is generated, the previous frame image, the previous frame stylized image and the optical flow image are input into a first stylized image generation model for stylized processing, a corresponding current frame stylized image is obtained, the first stylized image generation model is obtained by acquiring the first training sample set, for each group of training samples, the target object in the original image and the target object in the corresponding stylized image are subjected to the same motion simulation, an original image after motion simulation and a corresponding stylized image after motion simulation are obtained, a corresponding optical flow image is generated according to the original image and the corresponding original image after motion simulation, and the first stylized image generation model is trained according to the training sample, the optical flow image and the stylized image after motion simulation, so that the adjacent frames in the video are manufactured by motion simulation for model training, and the optical flow image between the original image and the original image after motion simulation is introduced as the input of the model, so that the first stylized image generation model can utilize the correlation between adjacent frames when generating a stylized image, the motion change between the generated stylized images of adjacent frames can be consistent with the motion change between adjacent frames, the unpredictable flicker existing between the inter-frame stylized images is avoided, and then the stability of the stylized video is improved.

[0148] It should be noted that, for the method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited to the order of the actions described, because according to the embodiments of the present application, certain steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of the present application.

[0149] Referring to Figure 5 , a structural block diagram of an embodiment of a model training device of the present application is shown, which can specifically include the following modules:

[0150] The sample set acquisition module 301 is configured to acquire a first training sample set, each group of training samples in the first training sample set includes an original image containing a target object and a corresponding stylized image;

[0151] The motion model module 302 is configured to perform the same motion simulation on the target object in the original image and the target object in the corresponding stylized image for each group of training samples, to obtain a motion-simulated original image and a corresponding motion-simulated stylized image.

[0152] The image generation module 303 is configured to generate a corresponding optical flow image according to the original image and the corresponding motion-simulated original image.

[0153] The model training module 304 is configured to train the first stylized image generation model according to the training samples, the optical flow image, and the motion-simulated stylized image.

[0154] In an optional embodiment of the present application, the motion simulation includes at least one of rotation, translation, random deformation, and motion blur.

[0155] In an optional embodiment of the present application, the model training module includes:

[0156] The image splicing sub-module is configured to perform image splicing on the original image, the stylized image, and the optical flow image in the channel dimension to obtain an input image.

[0157] The sample set composition sub-module is configured to compose a second training sample set according to the input image and the motion-simulated stylized image, each group of training samples in the second training sample set including one input image and a corresponding motion-simulated stylized image.

[0158] The model training sub-module is configured to train the first stylized image generation model according to the second training sample set.

[0159] In an optional embodiment of the present application, the model training sub-module includes:

[0160] The first adding unit is configured to weight and sum the image loss and the adversarial loss according to a first proportion of the image loss and the adversarial loss to obtain a first total loss, the difference between the first proportion and 1 being less than a first preset value.

[0161] The first training unit is configured to train the first stylized image generation model according to the second training sample set based on the first total loss until the first total loss converges.

[0162] The second adding unit is configured to weight and sum the image loss and the adversarial loss according to a second proportion of the image loss and the adversarial loss to obtain a second total loss after the first total loss converges, the second proportion being greater than or equal to a second preset value.

[0163] a second training unit configured to train the first stylized image generation model according to the second training sample set based on the second total loss until the second total loss converges.

[0164] According to an embodiment of the present application, by obtaining the first training sample set, each group of training samples in the first training sample set includes an original image containing a target object and a corresponding stylized image, for each group of training samples, the target object in the original image and the target object in the corresponding stylized image are subjected to the same motion simulation to obtain a motion simulated original image and a corresponding motion simulated stylized image, according to the original image and the corresponding motion simulated original image, a corresponding optical flow image is generated, according to the training sample, the optical flow image and the motion simulated stylized image, the first stylized image generation model is trained, so that the adjacent frames in the video are manufactured by motion simulation for model training, and the optical flow image between the original image and the motion simulated original image is introduced as the input of the model, so that the first stylized image generation model can utilize the correlation between adjacent frames when generating stylized images, and the motion change between the generated stylized images of adjacent frames can be consistent with the motion change between adjacent frames, avoiding unpredictable flicker existing between inter-frame stylized images, thereby improving the stability of the stylized video.

[0165] With reference to Figure 6 , a structural block diagram of an embodiment of an image stylization device of the present application is shown, which can specifically include the following modules:

[0166] The data acquisition module 401 is configured to acquire video data.

[0167] The first stylization processing module 402 is configured to input the previous frame image of the video data into the second stylized image generation model for stylization processing to obtain the corresponding previous frame stylized image, the second stylized image generation model is trained according to the first training sample set, and each group of training samples in the first training sample set includes an original image containing a target object and a corresponding stylized image.

[0168] The image generation module 403 is configured to generate a corresponding optical flow image according to the current frame image and the previous frame image of the video data.

[0169] The second stylization processing module 404 is configured to input the previous frame image, the previous frame stylized image and the optical flow image into a first stylized image generation model to perform stylization processing, to obtain a corresponding current frame stylized image, the first stylized image generation model being obtained by acquiring the first training sample set, performing the same motion simulation on the target object in the original image and the target object in the corresponding stylized image for each group of training samples, obtaining a motion simulated original image and a corresponding motion simulated stylized image, generating a corresponding optical flow image according to the original image and the corresponding motion simulated original image, and training the first stylized image generation model according to the training sample, the optical flow image and the motion simulated stylized image.

[0170] In an optional embodiment of the present application, the device further comprises:

[0171] The sample set composition module is configured to compose a third training sample set according to the current frame image and the corresponding current frame stylized image, each group of training samples in the third training sample set comprising a current frame image and a corresponding current frame stylized image.

[0172] The model training module is configured to train a third stylized image generation model according to the third training sample set.

[0173] The target video acquisition module is configured to acquire target video data.

[0174] The third stylization processing module is configured to input a frame image in the target video data into the third stylized image generation model to perform stylization processing, to obtain a corresponding stylized image.

[0175] According to an embodiment of the present invention, by acquiring video data, the previous frame image of the video data is input into a second stylized image generation model for stylization processing to obtain a corresponding stylized image of the previous frame. The second stylized image generation model is trained based on a first training sample set. Each group of training samples in the first training sample set includes an original image containing a target object and a corresponding stylized image. Based on the current frame image and the previous frame image of the video data, a corresponding optical flow image is generated. The previous frame image, the previous stylized image, and the optical flow image are input into the first stylized image generation model for stylization processing to obtain a corresponding stylized image of the current frame. The first stylized image generation model obtains the first training sample set and, for each group of training samples, stylizes the target object in the original image and the corresponding style. The same motion simulation is performed on the target object in the stylized image to obtain an original image after motion simulation and a corresponding stylized image after motion simulation. Based on the original image and the corresponding original image after motion simulation, a corresponding optical flow image is generated. The model is trained based on the training samples, the optical flow image, and the stylized image after motion simulation. This allows the adjacent frames in the video to be generated using motion simulation for model training. The optical flow image between the original image and the original image after motion simulation is introduced as the input to the model. Thus, the first stylized image generation model can utilize the correlation between adjacent frames when generating stylized images. The motion changes between the stylized images of adjacent frames can be consistent with the motion changes between adjacent frames, avoiding unpredictable flickering in the stylized images between frames, thereby improving the stability of the stylized video.

[0176] This invention also provides an electronic device, such as... Figure 7 As shown, it includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904. The processor 901, communication interface 902, and memory 903 communicate with each other via the communication bus 904.

[0177] Memory 903 is used to store computer programs;

[0178] When the processor 901 executes the program stored in the memory 903, it implements the steps described in any of the foregoing method embodiments.

[0179] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0180] The communication interface is used for communication between the terminal and other devices.

[0181] The memory can include a Random Access Memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0182] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processing (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0183] In another embodiment provided by the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores instructions, when the instructions are run on a computer, the computer is caused to execute the model training method in any of the above embodiments.

[0184] In another embodiment provided by the present application, a computer program product containing instructions is also provided, when the instructions are run on a computer, the computer is caused to execute the model training method in any of the above embodiments.

[0185] In another embodiment provided by the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores instructions, when the instructions are run on a computer, the computer is caused to execute the image stylization method in any of the above embodiments.

[0186] In yet another embodiment of the present disclosure, there is provided a computer program product comprising instructions which, when executed on a computer, cause the computer to carry out the image stylization method of any of the above-mentioned embodiments.

[0187] In the above-mentioned embodiments, the entire or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, the entire or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded into and executed by a computer, the entire or part of the processes or functions according to the embodiments of the present disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0188] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0189] The various embodiments in the specification are described in a related manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0190] The above only describes the preferred embodiments of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A model training method, characterized in that, include: Obtain a first training sample set, wherein each training sample in the first training sample set includes an original image containing the target object and a corresponding stylized image; The first training sample set is generated based on multiple video frames containing the first target object from the first video data; For each set of training samples, the same motion simulation is performed on the target object in the original image and the target object in the corresponding stylized image to obtain an original image after motion simulation and a corresponding stylized image after motion simulation; the motion simulation is to change the target object in the image to simulate various movements of the target object, including at least one of rotation, translation, random deformation, and motion blur; Based on the original image and the corresponding motion-simulated original image, a corresponding optical flow image is generated; Based on the training samples, optical flow images, and stylized images after motion simulation, a first stylized image generation model with an adversarial generative network architecture is trained, including: image stitching of the original image, stylized image, and optical flow image in the channel dimension to obtain the input image; A second training sample set is formed based on the input image and the stylized image after motion simulation. Each training sample in the second training sample set includes one input image and a corresponding stylized image after motion simulation. The model parameters of the first stylized image generation model are adjusted according to the first total loss and the second total loss corresponding to each training sample in the second training sample set. The first total loss and the second total loss are obtained by weighted summation of image loss and adversarial loss. The adversarial loss includes the loss of the true sample corresponding to the stylized image after motion simulation.

2. The method according to claim 1, characterized in that, The step of training the first stylized image generation model based on the second training sample set includes: Based on a first ratio of image loss and adversarial loss, the image loss and adversarial loss are weighted and summed to obtain a first total loss, wherein the difference between the first ratio and 1 is less than a first preset value. Based on the first total loss, the first stylized image generation model is trained according to the second training sample set until the first total loss converges. After the first total loss converges, the image loss and adversarial loss are weighted and summed according to the second ratio of the image loss and the adversarial loss to obtain the second total loss, wherein the second ratio is greater than or equal to a second preset value. Based on the second total loss, the first stylized image generation model is trained using the second training sample set until the second total loss converges.

3. An image stylization method, characterized in that, include: Acquire video data; The previous frame image of the video data is input into the second stylized image generation model for stylization processing to obtain the corresponding previous frame stylized image. The second stylized image generation model is trained based on the first training sample set. The first training sample set is generated based on multiple video frames containing the first target object in the first video data. Each training sample in the first training sample set includes an original image containing the target object and a corresponding stylized image. Based on the current frame image and the previous frame image of the video data, a corresponding optical flow image is generated; The previous frame image, the previous frame stylized image, and the optical flow image are input into a first stylized image generation model using an adversarial generative network architecture for stylization processing to obtain the corresponding current frame stylized image. The first stylized image generation model using an adversarial generative network architecture obtains the first training sample set, and for each set of training samples, performs the same motion simulation on the target object in the original image and the target object in the corresponding stylized image to obtain an original image after motion simulation and a corresponding stylized image after motion simulation. Based on the original image and the corresponding original image after motion simulation, a corresponding optical flow image is generated, and the model is trained based on the training samples, the optical flow image, and the stylized image after motion simulation. The motion simulation involves altering the target object in the image to simulate various movements of the target object, including at least one of rotation, translation, random deformation, and motion blur. The training based on the training samples, optical flow image, and motion-simulated stylized image includes: stitching the original image, stylized image, and optical flow image along the channel dimension to obtain an input image; forming a second training sample set based on the input image and the motion-simulated stylized image, wherein each training sample in the second training sample set includes one input image and a corresponding motion-simulated stylized image; adjusting the model parameters of the first stylized image generation model based on the first total loss and the second total loss corresponding to each training sample in the second training sample set; the first total loss and the second total loss are obtained by weighted summation of image loss and adversarial loss; the adversarial loss includes the loss of the true sample corresponding to the motion-simulated stylized image.

4. The method according to claim 3, characterized in that, The method further includes: A third training sample set is formed based on the current frame image and the corresponding current frame stylized image. Each training sample in the third training sample set includes a current frame image and a corresponding current frame stylized image. The third stylized image generation model is trained based on the third training sample set. Acquire target video data; The frame images in the target video data are input into the third stylized image generation model for stylization processing to obtain the corresponding stylized image.

5. A model training device, characterized in that, include: The sample set acquisition module is used to acquire a first training sample set, wherein each training sample in the first training sample set includes an original image containing the target object and a corresponding stylized image; the first training sample set is generated based on multiple video frames containing the first target object in the first video data; The motion model module is used to perform the same motion simulation on the target object in the original image and the target object in the corresponding stylized image for each group of training samples, so as to obtain an original image after motion simulation and a corresponding stylized image after motion simulation; the motion simulation is to change the target object in the image to simulate various motions of the target object, including at least one of rotation, translation, random deformation, and motion blur. The image generation module is used to generate a corresponding optical flow image based on the original image and the corresponding motion-simulated original image; The model training module is used to train a first stylized image generation model with an adversarial generative network architecture based on the training samples, optical flow images, and stylized images after motion simulation. The module includes: performing image stitching on the original image, stylized image, and optical flow image in the channel dimension to obtain the input image. A second training sample set is formed based on the input image and the stylized image after motion simulation. Each training sample in the second training sample set includes one input image and a corresponding stylized image after motion simulation. The model parameters of the first stylized image generation model are adjusted according to the first total loss and the second total loss corresponding to each training sample in the second training sample set. The first total loss and the second total loss are obtained by weighted summation of image loss and adversarial loss. The adversarial loss includes the loss of the true sample corresponding to the stylized image after motion simulation.

6. An image stylization device, characterized in that, include: The data acquisition module is used to acquire video data; The first stylization processing module is used to input the previous frame image of the video data into the second stylization image generation model for stylization processing to obtain the corresponding previous frame stylization image. The second stylization image generation model is trained based on the first training sample set. The first training sample set is generated based on multiple video frames containing the first target object in the first video data. Each training sample in the first training sample set includes an original image containing the target object and a corresponding stylization image. The image generation module is used to generate a corresponding optical flow image based on the current frame image and the previous frame image of the video data; The second stylization processing module is used to input the previous frame image, the previous frame stylized image, and the optical flow image into the first stylization image generation model using an adversarial generative network architecture for stylization processing to obtain the corresponding current frame stylized image. The first stylization image generation model using an adversarial generative network architecture obtains the first training sample set, and for each group of training samples, performs the same motion simulation on the target object in the original image and the target object in the corresponding stylized image to obtain an original image after motion simulation and a corresponding stylized image after motion simulation. Based on the original image and the corresponding original image after motion simulation, the corresponding optical flow image is generated, and the model is trained based on the training samples, the optical flow image, and the stylized image after motion simulation. The motion simulation involves altering the target object in the image to simulate various movements of the target object, including at least one of rotation, translation, random deformation, and motion blur. The training based on the training samples, optical flow image, and stylized image after motion simulation includes: stitching the original image, stylized image, and optical flow image along the channel dimension to obtain the input image; A second training sample set is formed based on the input image and the stylized image after motion simulation. Each training sample in the second training sample set includes one input image and a corresponding stylized image after motion simulation. The model parameters of the first stylized image generation model are adjusted according to the first total loss and the second total loss corresponding to each training sample in the second training sample set. The first total loss and the second total loss are obtained by weighted summation of image loss and adversarial loss. The adversarial loss includes the loss of the true sample corresponding to the stylized image after motion simulation.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-4.

Images