Stylized Image Generation Method, Apparatus, Electronic Device, and Storage Medium

Abstract

The embodiments of the present disclosure provide a method, device, electronic device and storage medium for generating a stylized image, the method comprising: determining a plurality of initial pairing data, and obtaining a style model to be used based on training of the initial pairing data; determining an original image to be processed from the original image based on preset screening conditions, and processing the original image to be processed based on the style model to be used to obtain a style image to be used; deforming the style image to be used to obtain a target style image, and using the original image to be processed and the target style image as stylized pairing data; training a stylized conversion model to be trained based on the stylized pairing data to obtain a target stylized conversion model. The technical solution provided by the embodiments of the present disclosure improves the pairing of the training data by introducing the style model to be used and deforming the style image to be used, so that the image output by the trained target stylized conversion model is more closely matched with the original image.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to a method, device, electronic device, and storage medium for generating a stylized image. Background Art

[0002] With the continuous development of image processing technology, users can use a variety of applications to process images so that the processed images can present the style they expect.

[0003] In the existing technology, after the relevant algorithms deployed on the server process the image, although images of the corresponding style type are obtained, the process of the server receiving the image, processing the image and then feeding back the processing results will bring about a large delay. The relevant algorithms with high computing resource requirements cannot be directly deployed on the client; at the same time, the obtained image may have defects. For example, some features in the image may not correspond to the features in the original image, which leads to poor image processing effect of the algorithm and reduces the user experience. Summary of the Invention

[0004] The embodiments of the present disclosure provide a method, apparatus, electronic device, and storage medium for generating stylized images. By introducing a style model to be used and deforming the style image to be used, the pairing of training data is improved, a lightweight model suitable for mobile terminals is trained based on the training data, and the delay of stylized image processing is reduced.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating a stylized image, the method comprising:

[0006] Determining a plurality of initial paired data, and training a style model to be used based on the plurality of initial paired data; wherein the initial paired data includes an original image and an initial style image processed by a 3D style generation model;

[0007] Determining a plurality of original images to be processed from the original images based on a preset screening condition, and processing the plurality of original images to be processed based on the style model to be used to obtain a style image to be used;

[0008] Obtaining target style images corresponding to the original images to be processed by deforming the style images to be used, and using the original images to be processed and the corresponding target style images as stylized paired data;

[0009] The stylized conversion model to be trained is trained based on the stylized paired data to obtain a target stylized conversion model, so that when a video frame to be processed is obtained, the video frame to be processed is stylized based on the target stylized conversion model to obtain a processed target video.

[0010] In a second aspect, an embodiment of the present disclosure further provides a stylized image generation device, the device comprising:

[0011] A module for determining a style model to be used, configured to determine a plurality of initial paired data and train a style model to be used based on the plurality of initial paired data; wherein the initial paired data includes an original image and an initial style image processed by a 3D style generation model;

[0012] a module for determining a style image to be used, configured to determine a plurality of original images to be processed from the original images based on a preset screening condition, and to process the plurality of original images to be processed based on the style model to be used to obtain the style image to be used;

[0013] a target style image determination module, configured to obtain a target style image corresponding to each original image to be processed by deforming the style image to be used, and use the original image to be processed and the corresponding target style image as stylized paired data;

[0014] A target stylized conversion model is used to train the stylized conversion model to be trained based on the stylized paired data to obtain a target stylized conversion model, so that when a video frame to be processed is obtained, the video frame to be processed is stylized based on the target stylized conversion model to obtain a processed target video.

[0015] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the stylized image generation method as described in any one of the embodiments of the present disclosure.

[0019] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute the stylized image generation method as described in any one of the embodiments of the present disclosure.

[0020] The technical solution of the embodiment of the present disclosure first determines multiple initial pairing data, and trains a style model to be used based on the multiple initial pairing data. Furthermore, based on preset screening conditions, multiple original images to be processed are determined from the original images, and the multiple original images to be processed are processed based on the style model to be used to obtain the style images to be used. By deforming the style images to be used, target style images corresponding to each original image to be processed are obtained, and the original images to be processed and the corresponding target style images are used as stylized pairing data. Finally, a stylized transfer model to be trained is trained based on the stylized pairing data to obtain a target stylized transfer model. When a video frame to be processed is obtained, the video frame to be processed is stylized based on the target stylized transfer model to obtain a processed target video. By introducing the style model to be used and deforming the style images to be used, the pairing property of the training data is improved. Furthermore, a lightweight model suitable for mobile terminals is trained based on the training data, avoiding the cumbersome process of image data transfer between the client and the server, reducing the delay of stylized image processing, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0022] Figure 1 A schematic diagram of a flow chart of a stylized image generation method provided in the first embodiment of the present disclosure;

[0023] Figure 2 This is a schematic diagram of generating stylized pairing data provided in the first embodiment of the present disclosure;

[0024] Figure 3 The original image, the style image to be used, and the target style image provided as examples in the first embodiment of the present disclosure;

[0025] Figure 4 This is a structural block diagram of a stylized image generation device provided in the second embodiment of the present disclosure;

[0026] Figure 5 This is a structural diagram of an electronic device provided in Example 3 of the present disclosure. DETAILED DESCRIPTION

[0027] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0028] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0029] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0030] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0032] Before introducing this technical solution, we will first provide an example of an application scenario. This disclosed technical solution can be applied to any scenario where special effects video generation is required. For example, while shooting a video through a related application, a corresponding pattern can be generated on a display interface based on the user's drawing operation, and then a special effects video containing the three-dimensional model corresponding to the pattern can be generated and displayed on the display interface.

[0033] Example 1

[0034] Figure 1This is a flow chart of a stylized image generation method provided in the first embodiment of the present disclosure. This embodiment is applicable to generating more compatible training data based on an existing image processing model, thereby training a model suitable for mobile devices with better image processing effects based on the obtained training data. The method can be performed by a stylized image generation device, which can be implemented in the form of software and / or hardware, and the hardware can be an electronic device such as a mobile terminal, PC, or server. The scene displayed in a special effects video is usually implemented by the cooperation of a client and a server. The method provided in this embodiment can be executed by the server, the client, or the cooperation of the client and the server.

[0035] like Figure 1 , the method of this embodiment includes:

[0036] S110: Determine a plurality of initial paired data, and train a style model to be used based on the plurality of initial paired data.

[0037] Among them, the device for executing the stylized image generation method provided by the embodiment of the present disclosure can be integrated into the application software that supports the special effect video processing function, and the software can be installed in the electronic device. Optionally, the electronic device can be a mobile terminal or a PC. The application software can be a type of software for image / video processing. The specific application software will not be described here one by one, as long as the image / video processing can be achieved. It can also be a specially developed application to implement the software for adding special effects and displaying the special effects, or it can be integrated in the corresponding page. The user can use the page integrated in the PC to process the video frame or specific image.

[0038] At the same time, since the application software can be installed and run on mobile terminals, the algorithm used by the application software to generate the target stylized image should be compatible with the mobile terminal. Specifically, compared with the server, the computing power and computing resources of the mobile terminal are relatively weak. Therefore, the algorithm deployed on the server and requiring high computing power cannot be run directly on the mobile terminal. Instead, a lightweight model adapted to the computing power of the mobile terminal needs to be retrained for the mobile platform to provide services to users with lower latency.

[0039] In this embodiment, to obtain a model suitable for mobile devices and with better image processing performance, it is first necessary to generate corresponding training data for the model using an existing 3D style generation model. Optionally, multiple original images containing facial information are obtained and each original image is input into a pre-trained 3D style generation model to obtain an initial style image after facial information processing.

[0040] Among them, the existing 3D style generation model can be a model deployed on the server side for generating an initial style image. In actual application, the 3D style generation model can be a model for generating images of the style type of 3D game characters. For the 3D style generation model, its input is the original image containing the user's facial features. It can be understood that the image at least contains the user's facial features, such as the user's daily photos or ID photos. Correspondingly, the output of the model is to make the user's face present an image of a specific 3D style type while retaining the original features of the user's facial features. It can be understood as a 3D facial image of a specific style type. When the model is a model for generating images of the style type of 3D game characters, in the output image, the user's face will present a visual effect similar to the character in the game. With the support of the 3D game character style special effects, the facial features are smoother, clearer, and more three-dimensional.

[0041] Those skilled in the art will appreciate that the training set used to train the 3D style generation model can include multiple, open-source, and diverse user facial images. For example, these images can include user portraits of various genders, age groups, and expressions, as well as from various viewing angles. These images can be obtained from open-source image databases. Furthermore, after the model is trained based on the training set, it can be deployed on a server associated with the application. This embodiment will not be further described here.

[0042] Based on this, the initial pairing data includes the original image and the initial style image processed by the 3D style generation model. It can be understood that in actual application, after the original image is processed by the 3D style generation model, the resulting image retains the original user's facial features while also making the user's facial image present a 3D game character style visual effect.

[0043] In this embodiment, the image output by the 3D style generation model may only partially match some features in the original image. For example, the two images may match features such as the user's gender and posture, but may not accurately match more subtle features in the user's facial image. For example, if the user in the original input image displays exaggerated facial expressions, resulting in significant facial movements, the corresponding features in the output initial style image may not accurately match the areas of the user's facial movements. Therefore, in order to provide better matching training data for the final model to be trained, it is necessary to train the first to-be-trained style model based on the initial matching data to obtain the style model to be used.

[0044] In this embodiment, the first style model to be trained can be a P2P model deployed on the server. Similar to the 3D style generation model, this model also takes as input an original image containing facial information and outputs an image of a specific style type, such as an image of a 3D game character. It will be appreciated that, given that the training data (i.e., initial paired data) already has a certain degree of matching, the images output by the trained style model to be used will have a higher degree of matching with the original images. The following describes the training process of the first style model to be trained.

[0045] Optionally, a first style model to be trained is obtained; for each initial paired data, the original image in the current initial paired data is used as the input of the first style model to be trained, and a first output image corresponding to the original image is obtained; based on the first output image and the initial style image corresponding to the original image, a loss value is determined to correct the model parameters in the first style model to be trained based on the loss value; and the convergence of the first loss function in the first style model to be trained is used as a training target to obtain the style model to be used.

[0046] Specifically, after obtaining multiple sets of initial paired data, the existing first style model to be trained can be used to process the large amount of initial paired data to generate a first output image, i.e., an image that has a higher degree of match with the original image input by the user. Furthermore, after obtaining the first output image, a loss value can be determined between the first output image and the initial style image. When using the loss value to correct the model parameters in the first style model to be trained, the training error of the first loss function in the first style model to be trained, i.e., the loss parameter, can be used as a criterion for detecting whether the first loss function has reached convergence. For example, whether the training error is less than a preset error, whether the error trend is stable, or whether the current number of iterations is equal to a preset number. If convergence conditions are met, such as whether the training error of the loss function is less than a preset error, or whether the error trend is stable, training of the first style model to be trained is complete, and iterative training can be terminated. If convergence conditions are not met, additional initial paired data can be obtained to continue model training until the training error of the loss function is within a preset range. It can be understood that when the training error of the loss function reaches convergence, a trained style model to be used can be obtained. At this time, after the original image including the user's facial information is input into the model, an image with a higher degree of match with the original image can be obtained.

[0047] S120 : Determine a plurality of original images to be processed from the original images based on a preset screening condition, and process the plurality of original images to be processed based on a style model to be used to obtain a style image to be used.

[0048] In this embodiment, when the style model to be used is a P2P model for generating 3D game character style type images, the image output by the model has a higher match with the original image than the initial style image, but the overall quality of the output image still has a certain gap with the original image. For example, the clarity of the output image is lower than the original image. Therefore, in order to obtain data with higher matching, multiple original images to be processed can be filtered out from the original image based on preset conditions, so that the style model to be used only processes the filtered images. The screening process of the original image is explained below.

[0049] Optionally, determine the original image to be processed in which the change angle of the part to be adjusted in each original image is greater than a preset change angle threshold. Specifically, the part to be adjusted includes the facial features, for example, the user's eyes, nose, ears and mouth, etc. The preset angle can be a change angle set separately for the user's facial features, for example, the angle between the upper and lower contours of the eyes, or the angle set separately for the degree of curvature of the mouth. In actual application, relevant image processing software can be used to determine the angles of the user's facial features in the original image, and then compare these angles with the preset angles. When it is determined that the angle of the facial features is greater than the corresponding preset angle, the original image is determined as the original image to be used. In actual application, after determining the original image to be used, in order to facilitate the use of subsequent models, these images can be copied to a specific database, or these images can be marked with specific labels. The embodiments of the present disclosure are not specifically limited here.

[0050] In this embodiment, when an angle irrelevant to the user's daily expression is used as the preset change angle, the original image to be processed is an image in which the expression change amplitude is at least greater than the user's daily expression. It can be understood as a facial image corresponding to when the user makes larger facial movements and more exaggerated expressions.

[0051] After selecting the original images to be used, each of the processed original images is then fed into the style model to generate a corresponding style image. Since the input to the style model is a facial image corresponding to a user with large facial movements and exaggerated expressions, which differs from the input to the initial 3D style generation model, the outputs from the two models are also different. This means that the features corresponding to the user's facial features differ significantly between the two output images.

[0052] For example, after obtaining images in which the user produces large facial movements as the original images to be used, these images can be input into a P2P model deployed on the server side for generating 3D game character style images for processing, thereby obtaining 3D game character style images that have a higher degree of match with the original images.

[0053] S130 , obtaining target style images corresponding to the original images to be processed by deforming the style images to be used, and using the original images to be processed and the corresponding target style images as stylized paired data.

[0054] In this embodiment, while the resulting style image to be used has a high degree of match with the original image to be used, features corresponding to areas of the image where the user makes significant facial movements still do not accurately match the corresponding areas in the original image. For example, in the original image to be used, the corners of the user's mouth are significantly raised. While the resulting style image to be used creates a 3D game character-like visual effect, the corners of the mouth do not rise as significantly as in the original image to be used. Therefore, in order to construct higher-quality training samples for the model ultimately used by the application to provide services to users, the style image to be used needs to be deformed. In actual applications, the style image to be used can be processed using a thin plate spline transform (TPS). TPS is a non-rigid deformation algorithm whose input is multiple pairs of matching points of the same parts in the two images, for example, the matching points of the user's mouth in the original image to be processed and the user's mouth in the style image to be used. The corresponding output is the mapping coordinates of the same parts in the two images. The deformation processing process is described in detail below.

[0055] Optionally, pixel information of key points in the original image to be processed and the style image to be used is determined; based on the pixel information, deformation parameters are determined, so as to attach the part to be adjusted in the original image to be processed to the style image to be used based on the deformation parameters to obtain the target style image.

[0056] Specifically, since the goal of TPS is to solve a function f such that f(P i )=P′ i (1≤i≤n), and the bending energy function is minimized. At this point, the points on the style image to be used can be well corrected by interpolation. When the deformation function is used as the process of bending a steel plate, the energy required to make the steel plate pass through a given n points can be expressed as:

[0057]

[0058] Among them, (x, y) is the arbitrary coordinate of the key point in the original image to be processed. Furthermore, it can be proved that the interpolation function of TPS is the function with the minimum bending energy:

[0059]

[0060] By deduction, we can get the coordinates (x, y) of any key point in the original image to be processed to the coordinates (x, y) of any key point after deformation. i ,y i )’s mapping:

[0061]

[0062] Among them, w is the coefficient matrix and U is the basis function.

[0063] For example, after obtaining the style image to be used, the first step is to determine the pixel information of key points in the original image to be processed and the style image to be used. Specifically, in both images, the pixel information corresponding to the portion of the mouth that produces a larger upward angle is determined. Furthermore, the deformation parameters of the mouth are determined based on the aforementioned formula. Based on these deformation parameters, the mouth portion of the original image to be processed is attached to the corresponding position of the mouth portion in the style image to be used, thereby replacing the original portion of the style image to be used that does not match the user's actual mouth features.

[0064] It can be understood that the style image to be used after the key point part is attached and adjusted is the target style image. The target style image not only retains the unique facial features of the user in the original image to be processed, but also makes the image present a 3D game character style type visual effect, and has a high degree of matching with the original image to be processed.

[0065] In this embodiment, once the target style image is obtained, it can be combined with the corresponding original image to be processed to obtain stylized paired data. The stylized paired data is the data used to train the model actually used in the application, for example, the data used to train the image processing model to be deployed on the mobile terminal.

[0066] In order to explain the process of determining stylized paired data in a more holistic way, Figure 2 This process is explained.

[0067] See also Figure 2After obtaining the original image containing the user's facial information, the original image can be processed using a model deployed on the server for generating 3D game character style images. The processed results are then combined with the original image to obtain initial paired training data. Based on this training data, a P2P model to be trained, also used to generate 3D game character style images, can be trained. It is understood that after the P2P model is trained, its output image has a higher matching probability with the original image. However, since the overall quality of the output image is lower than that of the original image, it is necessary to screen out the original images in which the user displays large facial expressions or has large angular changes in certain facial parts. These original images are then input into the trained P2P model to obtain the corresponding style images to be used, i.e., 3D game character style images corresponding to images with large facial expressions and large angular changes in certain facial parts. In these images, the parts with large facial expressions and large angular changes in certain facial parts still differ from the corresponding parts in the original images to be used.

[0068] Continue to see Figure 2 In order to improve the matching of the style image to be used and the corresponding original image, it is also necessary to deform the parts of the style image to be used that have large angle changes with the user's large expressions or certain parts of the face based on the TPS technology. For example, when the corners of the user's mouth in the original image to be processed are greatly raised, the corresponding parts in the style image to be used can be deformed based on the TPS technology. After obtaining the deformation parameters, the mouth area in the original image can be attached to the image to be used, thereby obtaining a target style image with a high matching property with the original image to be processed. Finally, the target style image is combined with the corresponding original image to be processed, that is, the final version of paired training data is constructed. These training data with high matching property can be used to train the target stylized conversion model to be used in the application.

[0069] It should be noted that, in actual application, the image processing model deployed on the mobile terminal can be a lightweight stylized conversion model for generating 3D game character style images. It can be understood that after the image including the user's facial information is input into the stylized conversion model for processing, a 3D game character style image with a high degree of match with the user's facial features can be generated with lower computing resources.

[0070] Based on the above description, it can be determined that before using stylized paired data to train the stylized transfer model, it is necessary to first determine the stylized transfer model to be trained of the target grid structure; assemble the discriminator to be trained for the stylized transfer model to be trained, and set parameter adjustment constraints for the discriminator to be trained, so as to constrain the model parameters in the stylized transfer model to be trained and the discriminator to be trained based on the constraints to obtain the target stylized transfer model.

[0071] In actual applications, the stylized transfer model to be trained can be a model based on a generative adversarial network (GAN). The GAN can be associated with a generator and a discriminator. The generator randomly samples from the latent space as input, and its output needs to imitate the real samples in the training set as much as possible. The input of the discriminator is the real samples and the output of the generator. Based on this, it can be understood that the stylized transfer model to be trained in this embodiment can also be spliced with a discriminator to be trained, so that in the subsequent process, the stylized transfer model with modified parameters can regenerate a target stylized image.

[0072] It should be noted that in order to make the stylized model to be trained compatible with the computing power of the mobile terminal, multiple lightweight models can be developed in advance, that is, models with fewer network layers and less refined model parameters. After these models are deployed on the mobile terminal, the original image can be processed. Since these models can generate images with a lower signal-to-noise ratio based on a lower number of channels, the image processing latency can be reduced. After all models output processing results corresponding to the original image, based on factors such as image processing effect and model running time, a model with better image processing effect and shorter running time can be selected from multiple models as the stylized conversion model to be trained.

[0073] In this embodiment, a common P2P model easily generates a large amount of erroneous data when trained in a 3D game character style. This is due to the instability of the discriminator to be trained. Therefore, to improve the stability of the subsequent model training process, parameter adjustment constraints need to be set for the discriminator to be trained. For example, Spectral Norm Regularization (SNR) is used. This technique introduces regularization constraints from the perspective of the spectral norm of the parameter matrix of each layer of the neural network, making the neural network less sensitive to input perturbations, thereby making the subsequent training process more stable and easier to converge.

[0074] Specifically, when the parameter adjustment constraint condition set for the discriminator to be trained is SNR, the discriminator to be trained can satisfy the Lipschitz continuity in the subsequent training process, that is,

[0075]

[0076] Where K is a constant. For the function f, the smallest constant K that satisfies the above conditions is the Lipschitz constant. In this embodiment, SNR performs a singular value decomposition (SVD) on the parameters of each layer of the neural network. After decomposition, the largest singular value is limited to 1. That is, after each update of the network parameters, they are divided by the largest singular value. On this basis, the stretch value of each layer of the discriminator network will not exceed 1, so that the discriminator to be trained satisfies Lipschitz continuity during the training process, and the training process of the discriminator to be trained is more stable and easier to converge.

[0077] S140 , training the stylized transfer model to be trained based on the stylized paired data to obtain a target stylized transfer model, so that when a video frame to be processed is obtained, stylized processing is performed on the video frame to be processed based on the target stylized transfer model to obtain a processed target video.

[0078] In this embodiment, the training process of the stylized transfer model to be trained can be as follows: inputting the original image to be processed in the stylized paired data into the style transfer model to be trained to obtain a second actual output image; inputting the second actual output image and the target style image in the stylized paired data into the discriminator to be trained to obtain a discrimination result; adjusting the model parameters in the style transfer model to be trained and the discriminator to be trained based on the discrimination result and constraints; and taking the convergence of the loss functions in the style transfer model to be trained and the discriminator to be trained as a training goal to obtain a target stylized transfer model.

[0079] Specifically, after obtaining multiple sets of stylized paired data, the existing stylized transfer model to be trained can be used to process the large amount of stylized paired data to generate a second actual output image, i.e., an image that has a higher degree of match with the original image input by the user. Furthermore, after obtaining the second actual output image, a loss value can be determined between the second actual output image and the target style image in the stylized paired data based on the trained discriminator. This loss value can be understood as the discrimination result. When using the loss value to correct the model parameters in the trained style transfer model and the trained discriminator under a pre-set spectral norm regularization constraint, the training error of the loss function in the trained style transfer model and the trained discriminator, i.e., the loss parameter, can be used as a criterion for detecting whether the loss function has reached convergence. For example, whether the training error is less than a preset error, whether the error trend is stable, or whether the current number of iterations is equal to a preset number. If convergence conditions are met, such as if the training error of the loss function is less than a preset error, or if the error trend is stable, the training of the trained style transfer model and the trained discriminator is complete, and iterative training can be terminated. If convergence conditions are not met, further stylized paired data can be obtained to continue training the model and discriminator until the training error of the loss function is within a preset range. It can be understood that when the training error of the loss function reaches convergence, the trained target stylized conversion model is obtained. At this point, by inputting the original image containing the user's facial information into the model, a visual effect that retains the user's original facial features while presenting the style of the 3D game character can be achieved.

[0080] It should be noted that after obtaining the target stylization model, the target stylization model can also be deployed to the client, so that when the video frames to be processed are obtained, the video frames to be processed can be stylized based on the target stylization model to obtain the target video frames, so as to obtain the target video based on each target video frame.

[0081] Specifically, since the client can be installed in the user's mobile terminal device, and the mobile terminal device generally has the function of collecting user facial images, after the target stylized model is deployed on the client and it is detected that the user triggers the control associated with generating the target stylized model, the facial information in the video collected by the front camera or rear camera of the mobile terminal can be recognized. After the user's facial information is recognized, the corresponding video clip can be divided into multiple video frames, or the video manually imported into the application by the user can be directly divided into multiple video frames.

[0082] Furthermore, after acquiring multiple video frames, these frames can be processed based on the target stylization model to obtain the target video frame. It can be understood that when the target stylization model generates an image for the user in the style of a 3D game character, the target video frame retains the user's facial information while also presenting the visual effects of a 3D game character. Finally, the processed video frames are spliced together to obtain the target video. By displaying the target video on the display interface associated with the application, it can be presented to the user, enhancing the interest of the video content.

[0083] It should be noted that since the trained target stylization model can be directly deployed on the mobile terminal, when processing the user's facial image, the image / video can be processed directly on the mobile terminal, avoiding the tedious process of uploading the image / video to the server and then receiving the processing results from the server. It reduces the lag and delay when the application software presents the target video to the user, making the video playback with the target stylized image smoother, thereby improving the user experience.

[0084] In this embodiment, in order to more clearly reflect the differences between the original image, the style image to be used output by the style model to be used, and the target style image output by the target stylized model, it is also possible to use Figure 3 Take this as an example to illustrate.

[0085] See also Figure 3 When a user captures an image containing facial information using a mobile device's camera, this image can be used as the original image, i.e., the first image containing the user's open mouth expression. Furthermore, the original image is fed into the trained style model to be used on the server, resulting in a style image to be used in the style of a 3D game character. As can be seen from the figure, while this image generally reflects some of the user's facial features and presents a 3D game character-style visual effect, it does not show the user's open mouth expression. In other words, the corresponding part of the mouth in the style image to be used is inconsistent with that in the original image.

[0086] Continue to see Figure 3 After the target stylization model is trained based on the solution of this embodiment and deployed on the mobile terminal to process the original image, the target style image can be output. As can be seen from the figure, the target style image not only reflects some features of the user's face as a whole, like the style image to be used, and presents a visual effect in the style of a 3D game character, but also reproduces the expression of the user's open mouth in the image, so that the output image maintains a high degree of consistency with the original image in key features such as facial features, eliminating the problem of poor matching between the image obtained based on the style model to be used and the original image.

[0087] The technical solution of this embodiment first determines multiple initial pairing data and trains a style model to be used based on the multiple initial pairing data. Furthermore, based on preset screening conditions, multiple original images to be processed are determined from the original images, and the multiple original images to be processed are processed based on the style model to be used to obtain the style images to be used. Target style images corresponding to each original image to be processed are obtained by deforming the style images to be used, and the original images to be processed and the corresponding target style images are used as stylized pairing data. Finally, a stylized transfer model to be trained is trained based on the stylized pairing data to obtain a target stylized transfer model. When a video frame to be processed is obtained, the video frame to be processed is stylized based on the target stylized transfer model to obtain a processed target video. By introducing the style model to be used and deforming the style images to be used, the pairing quality of the final training data is improved. Furthermore, a lightweight model suitable for mobile terminals is trained based on the training data, avoiding the cumbersome process of image data transfer between the client and the server, reducing the delay of stylized image processing, and improving the user experience.

[0088] Example 2

[0089] Figure 4 This is a structural block diagram of a stylized image generation device provided in the second embodiment of the present disclosure, which can execute the stylized image generation method provided in any embodiment of the present disclosure and has the corresponding functional modules and beneficial effects of the execution method. Figure 4 As shown, the apparatus specifically includes: a to-be-used style model determination module 210 , a to-be-used style image determination module 220 , a target style image determination module 230 , and a target stylization conversion model determination module 240 .

[0090] The style model to be used determination module 210 is configured to determine a plurality of initial paired data and train a style model to be used based on the plurality of initial paired data; wherein the initial paired data includes an original image and an initial style image processed by a 3D style generation model.

[0091] The style image to be used determining module 220 is configured to determine a plurality of original images to be processed from the original images based on a preset screening condition, and process the plurality of original images to be processed based on the style model to be used to obtain the style image to be used.

[0092] The target style image determination module 230 is configured to obtain a target style image corresponding to each original image to be processed by deforming the style image to be used, and use the original image to be processed and the corresponding target style image as stylized paired data.

[0093] The target stylization conversion model determination module 240 is used to train the stylization conversion model to be trained based on the stylization pairing data to obtain a target stylization conversion model, so that when a video frame to be processed is obtained, the video frame to be processed is stylized based on the target stylization conversion model to obtain a processed target video.

[0094] On the basis of the above technical solutions, the to-be-used style model determination module 210 includes an original image acquisition unit and an initial style image generation unit.

[0095] The original image acquisition unit is used to acquire multiple original images including facial information.

[0096] The initial style image generation unit is used to input each original image into a pre-trained 3D style generation model to obtain an initial style image after processing the facial information.

[0097] On the basis of the above technical solutions, the module 210 for determining the style model to be used further includes a style model to be trained acquiring unit, a first output image generating unit, a loss value determining unit, and a style model to be used determining unit.

[0098] The to-be-trained style model acquisition unit is configured to acquire a first to-be-trained style model.

[0099] The first output image generating unit is configured to, for each initial paired data, use the original image in the current initial paired data as the input of the first style model to be trained to obtain a first output image corresponding to the original image.

[0100] A loss value determining unit is configured to determine a loss value based on the first output image and an initial style image corresponding to the original image, so as to correct model parameters in the first style model to be trained based on the loss value.

[0101] The to-be-used style model determining unit is configured to take the convergence of the first loss function in the first to-be-trained style model as a training target to obtain the to-be-used style model.

[0102] On the basis of the above technical solutions, the preset screening condition includes that the change angle of the part to be adjusted is greater than a preset change angle threshold.

[0103] Optionally, the style image determination module 220 is further configured to determine the original image to be processed in which the change angle of the part to be adjusted in each original image is greater than a preset change angle threshold; wherein the part to be adjusted includes facial features.

[0104] Optionally, the style image to be used determination module 220 is further used to input each original image to be processed into the style model to be used to obtain a style image to be used corresponding to each original image; wherein the style image to be used has different features from the initial style image corresponding to the original image to be processed.

[0105] Based on the above technical solutions, the target style image determination module 230 includes a pixel information determination unit and a target style image generation unit.

[0106] The pixel information determining unit is used to determine the pixel information of key points in the original image to be processed and the style image to be used.

[0107] The target style image generating unit is configured to determine deformation parameters based on the pixel information, so as to attach the part to be adjusted in the original image to be processed to the style image to be used based on the deformation parameters to obtain the target style image.

[0108] Based on the above technical solutions, the stylized image generation device further includes a parameter adjustment constraint condition setting module.

[0109] A parameter adjustment constraint setting module is used to determine a stylized conversion model to be trained for a target grid structure; assemble a discriminator to be trained for the stylized conversion model to be trained, and set parameter adjustment constraints for the discriminator to be trained, so as to constrain the model parameters in the stylized conversion model to be trained and the discriminator to be trained based on the constraints to obtain the target stylized conversion model.

[0110] On the basis of the above technical solutions, the target stylization conversion model determination module 240 includes a second actual output image generation unit, a discrimination result generation unit, a parameter adjustment unit and a target stylization conversion model determination unit.

[0111] The second actual output image generating unit is configured to input the original image to be processed in the stylized paired data into the style conversion model to be trained to obtain a second actual output image.

[0112] A discrimination result generating unit is configured to input the second actual output image and the target style image in the stylized paired data into the discriminator to be trained to obtain a discrimination result.

[0113] A parameter adjustment unit is used to adjust the model parameters in the style conversion model to be trained and the discriminator to be trained based on the discrimination result and the constraint condition.

[0114] The target stylized transfer model determination unit is configured to take the convergence of the loss functions of the to-be-trained style transfer model and the to-be-trained discriminator as a training target to obtain the target stylized transfer model.

[0115] Based on the above technical solutions, the stylized image generation device also includes a model deployment module.

[0116] The model deployment module is used to deploy the target stylization model to the client, so that when the video frames to be processed are obtained, the video frames to be processed are stylized based on the target stylization model to obtain target video frames, so as to obtain target videos based on each target video frame.

[0117] The technical solution provided by this embodiment first determines multiple initial pairing data and trains a style model to be used based on the multiple initial pairing data. Furthermore, based on preset screening conditions, multiple original images to be processed are determined from the original images, and the multiple original images to be processed are processed based on the style model to be used to obtain the style images to be used. Target style images corresponding to each original image to be processed are obtained by deforming the style images to be used, and the original images to be processed and the corresponding target style images are used as stylized pairing data. Finally, a stylized transfer model to be trained is trained based on the stylized pairing data to obtain a target stylized transfer model. When a video frame to be processed is obtained, the video frame to be processed is stylized based on the target stylized transfer model to obtain a processed target video. By introducing the style model to be used and deforming the style images to be used, the pairing quality of the training data is improved. Furthermore, a lightweight model suitable for mobile terminals is trained based on the training data, avoiding the cumbersome process of image data transfer between the client and the server, reducing the delay of stylized image processing, and improving the user experience.

[0118] The stylized image generation device provided in the embodiments of the present disclosure can execute the stylized image generation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0119] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0120] Example 3

[0121] Figure 5 This is a structural diagram of an electronic device provided by the third embodiment of the present disclosure. Figure 5, which shows an electronic device (eg Figure 5 The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0122] like Figure 5 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a pattern processor, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 306 into a random access memory (RAM) 303. Various programs and data required for the operation of the electronic device 300 are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An edit / output (I / O) interface 305 is also connected to the bus 304.

[0123] Typically, the following devices may be connected to the I / O interface 305: an editing device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 5 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0124] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 306, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0125] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0126] The electronic device provided in the embodiment of the present disclosure and the stylized image generation method provided in the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0127] Example 4

[0128] An embodiment of the present disclosure provides a computer storage medium having a computer program stored thereon. When the program is executed by a processor, the stylized image generation method provided in the above embodiment is implemented.

[0129] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0130] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0131] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0132] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0133] Determining a plurality of initial paired data, and training a style model to be used based on the plurality of initial paired data; wherein the initial paired data includes an original image and an initial style image processed by a 3D style generation model;

[0134] Determining a plurality of original images to be processed from the original images based on a preset screening condition, and processing the plurality of original images to be processed based on the style model to be used to obtain a style image to be used;

[0135] Obtaining target style images corresponding to the original images to be processed by deforming the style images to be used, and using the original images to be processed and the corresponding target style images as stylized paired data;

[0136] The stylized conversion model to be trained is trained based on the stylized paired data to obtain a target stylized conversion model, so that when a video frame to be processed is obtained, the video frame to be processed is stylized based on the target stylized conversion model to obtain a processed target video.

[0137] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0139] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0140] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0141] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0142] According to one or more embodiments of the present disclosure, [Example 1] provides a method for generating a stylized image, the method comprising:

[0143] Determining a plurality of initial paired data, and training a style model to be used based on the plurality of initial paired data; wherein the initial paired data includes an original image and an initial style image processed by a 3D style generation model;

[0144] Determining a plurality of original images to be processed from the original images based on a preset screening condition, and processing the plurality of original images to be processed based on the style model to be used to obtain a style image to be used;

[0145] Obtaining target style images corresponding to the original images to be processed by deforming the style images to be used, and using the original images to be processed and the corresponding target style images as stylized paired data;

[0146] The stylized conversion model to be trained is trained based on the stylized paired data to obtain a target stylized conversion model, so that when a video frame to be processed is obtained, the video frame to be processed is stylized based on the target stylized conversion model to obtain a processed target video.

[0147] According to one or more embodiments of the present disclosure, [Example 2] provides a method for generating a stylized image, further comprising:

[0148] Optionally, obtaining a plurality of original images including facial information;

[0149] Each original image is input into the pre-trained 3D style generation model to obtain the initial style image after processing the facial information.

[0150] According to one or more embodiments of the present disclosure, [Example 3] provides a method for generating a stylized image, further comprising:

[0151] Optionally, obtain a first style model to be trained;

[0152] For each initial paired data, using the original image in the current initial paired data as input of the first to-be-trained style model to obtain a first output image corresponding to the original image;

[0153] determining a loss value based on the first output image and an initial style image corresponding to the original image, so as to modify model parameters in the first to-be-trained style model based on the loss value;

[0154] The style model to be used is obtained by taking the convergence of the first loss function in the first style model to be trained as a training goal.

[0155] According to one or more embodiments of the present disclosure, [Example 4] provides a method for generating a stylized image, further comprising:

[0156] Optionally, the preset screening condition includes that a change angle of the part to be adjusted is greater than a preset change angle threshold, and determining a plurality of original images to be processed from the original image based on the preset screening condition includes:

[0157] Determine the original images to be processed in which the change angle of the part to be adjusted in each original image is greater than a preset change angle threshold;

[0158] Among them, the parts to be adjusted include the facial features.

[0159] According to one or more embodiments of the present disclosure, [Example 5] provides a method for generating a stylized image, further comprising:

[0160] Optionally, each original image to be processed is input into the style model to be used to obtain a style image to be used corresponding to each original image;

[0161] The style image to be used has different features from an initial style image corresponding to the original image to be processed.

[0162] According to one or more embodiments of the present disclosure, [Example 6] provides a method for generating a stylized image, further comprising:

[0163] Optionally, determining pixel information of key points in the original image to be processed and the style image to be used;

[0164] Based on the pixel information, a deformation parameter is determined, so as to attach the part to be adjusted in the original image to be processed to the style image to be used based on the deformation parameter to obtain a target style image.

[0165] According to one or more embodiments of the present disclosure, [Example 7] provides a method for generating a stylized image, further comprising:

[0166] Optionally, determining a stylized transfer model to be trained for the target grid structure;

[0167] The stylized transfer model to be trained is assembled with a discriminator to be trained, and parameter adjustment constraints are set for the discriminator to be trained, so as to perform constraint adjustment on model parameters in the stylized transfer model to be trained and the discriminator to be trained based on the constraints, so as to obtain the target stylized transfer model.

[0168] According to one or more embodiments of the present disclosure, [Example 8] provides a method for generating a stylized image, further comprising:

[0169] Optionally, the original image to be processed in the stylized paired data is input into the style conversion model to be trained to obtain a second actual output image;

[0170] Inputting the second actual output image and the target style image in the stylized paired data into the discriminator to be trained to obtain a discrimination result;

[0171] Adjusting model parameters in the to-be-trained style conversion model and the to-be-trained discriminator based on the discrimination result and the constraint condition;

[0172] The convergence of the loss functions in the to-be-trained style transfer model and the to-be-trained discriminator is used as a training goal to obtain the target stylized transfer model.

[0173] According to one or more embodiments of the present disclosure, [Example 9] provides a method for generating a stylized image, further comprising:

[0174] Optionally, the target stylization model is deployed to the client, so that when a video frame to be processed is obtained, the video frame to be processed is stylized based on the target stylization model to obtain a target video frame, so as to obtain a target video based on each target video frame.

[0175] According to one or more embodiments of the present disclosure, [Example 10] provides a stylized image generation device, including:

[0176] A module for determining a style model to be used, configured to determine a plurality of initial paired data and train a style model to be used based on the plurality of initial paired data; wherein the initial paired data includes an original image and an initial style image processed by a 3D style generation model;

[0177] a module for determining a style image to be used, configured to determine a plurality of original images to be processed from the original images based on a preset screening condition, and to process the plurality of original images to be processed based on the style model to be used to obtain the style image to be used;

[0178] a target style image determination module, configured to obtain a target style image corresponding to each original image to be processed by deforming the style image to be used, and use the original image to be processed and the corresponding target style image as stylized paired data;

[0179] A target stylized conversion model is used to train the stylized conversion model to be trained based on the stylized paired data to obtain a target stylized conversion model, so that when a video frame to be processed is obtained, the video frame to be processed is stylized based on the target stylized conversion model to obtain a processed target video.

[0180] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0181] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0182] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for generating a stylized image, characterized in that: include: Determining a plurality of initial paired data, and training a style model to be used based on the plurality of initial paired data; wherein the initial paired data includes an original image and an initial style image processed by a 3D style generation model; Determining a plurality of original images to be processed from the original images based on a preset screening condition, and processing the plurality of original images to be processed based on the style model to be used to obtain a style image to be used; Obtaining target style images corresponding to the original images to be processed by deforming the style images to be used, and using the original images to be processed and the corresponding target style images as stylized paired data; Training a to-be-trained stylized conversion model based on the stylized paired data to obtain a target stylized conversion model, so that when a to-be-processed video frame is acquired, stylizing the to-be-processed video frame based on the target stylized conversion model to obtain a processed target video; The step of deforming the style image to be used to obtain a target style image corresponding to each original image to be processed includes: Determining pixel information of key points in the original image to be processed and the style image to be used; Based on the pixel information, a deformation parameter is determined, so as to attach the part to be adjusted in the original image to be processed to the style image to be used based on the deformation parameter to obtain a target style image.

2. The method according to claim 1, characterized in that The determining of a plurality of initial pairing data includes: Acquire multiple original images including facial information; Each original image is input into the pre-trained 3D style generation model to obtain the initial style image after processing the facial information.

3. The method according to claim 1, characterized in that The step of training the style model to be used based on the plurality of initial paired data includes: Get the first style model to be trained; For each initial paired data, using the original image in the current initial paired data as input of the first to-be-trained style model to obtain a first output image corresponding to the original image; determining a loss value based on the first output image and an initial style image corresponding to the original image, so as to modify model parameters in the first to-be-trained style model based on the loss value; The style model to be used is obtained by taking the convergence of the first loss function in the first style model to be trained as a training goal.

4. The method according to claim 1, wherein The preset screening condition includes that the change angle of the part to be adjusted is greater than a preset change angle threshold, and the step of determining a plurality of original images to be processed from the original images based on the preset screening condition includes: Determine the original images to be processed in which the change angle of the part to be adjusted in each original image is greater than a preset change angle threshold; Among them, the parts to be adjusted include the facial features.

5. The method according to claim 1, characterized in that The processing of the plurality of original images to be processed based on the style model to be used to obtain the style image to be used includes: Inputting each original image to be processed into the style model to be used to obtain a style image to be used corresponding to each original image; The style image to be used has different features from an initial style image corresponding to the original image to be processed.

6. The method according to claim 1, characterized in that Before training the to-be-trained stylized conversion model based on the stylized paired data to obtain a target stylized conversion model, the method further includes: Determine the stylized transfer model to be trained for the target grid structure; The stylized transfer model to be trained is assembled with a discriminator to be trained, and parameter adjustment constraints are set for the discriminator to be trained, so as to perform constraint adjustment on model parameters in the stylized transfer model to be trained and the discriminator to be trained based on the constraints, so as to obtain the target stylized transfer model.

7. The method according to claim 6, characterized in that The step of training the stylized conversion model to be trained based on the stylized paired data to obtain a target stylized conversion model includes: Inputting the original image to be processed in the stylized paired data into the style conversion model to be trained to obtain a second actual output image; Inputting the second actual output image and the target style image in the stylized paired data into the discriminator to be trained to obtain a discrimination result; Adjusting model parameters in the to-be-trained style conversion model and the to-be-trained discriminator based on the discrimination result and the constraint condition; The convergence of the loss functions in the to-be-trained style transfer model and the to-be-trained discriminator is used as a training goal to obtain the target stylized transfer model.

8. The method according to claim 1, characterized in that Also includes: The target stylization model is deployed to the client, so that when a video frame to be processed is obtained, the video frame to be processed is stylized based on the target stylization model to obtain a target video frame, so as to obtain a target video based on each target video frame.

9. A stylized image generation device, characterized in that: include: A module for determining a style model to be used, configured to determine a plurality of initial paired data and train a style model to be used based on the plurality of initial paired data; wherein the initial paired data includes an original image and an initial style image processed by a 3D style generation model; a module for determining a style image to be used, configured to determine a plurality of original images to be processed from the original images based on a preset screening condition, and to process the plurality of original images to be processed based on the style model to be used to obtain the style image to be used; a target style image determination module, configured to obtain a target style image corresponding to each original image to be processed by deforming the style image to be used, and use the original image to be processed and the corresponding target style image as stylized paired data; a target stylization conversion model, configured to train the stylization conversion model to be trained based on the stylization paired data to obtain a target stylization conversion model, so as to perform stylization processing on the video frame to be processed based on the target stylization conversion model when a video frame to be processed is obtained, to obtain a processed target video; The target style image determination module includes a pixel information determination unit and a target style image generation unit; The pixel information determining unit is used to determine pixel information of key points in the original image to be processed and the style image to be used; The target style image generating unit is configured to determine deformation parameters based on the pixel information, so as to attach the portion to be adjusted in the original image to be processed to the style image to be used based on the deformation parameters to obtain the target style image.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the stylized image generation method according to any one of claims 1 to 8.

11. A storage medium comprising computer executable instructions, wherein when the computer executable instructions are executed by a computer processor, the computer executable instructions are used to perform the stylized image generation method according to any one of claims 1 to 8.

Images