Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed image; performing initial special effect processing on the to-be-processed image to obtain an initial special effect image; based on a target generation model, taking the initial special effect image as input, extracting target image information of the initial special effect image, and based on target style special effect information, generating a target special effect image whose semantics and the target image information conform to a preset condition, the target generation model being used for generating a target style image, therefore, the special effect generation efficiency and the special effect are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Currently, a variety of special effects tools are provided in short video platforms, such as 3D special effects. Users can add various special effects to short videos, making the video content richer and more interesting. In related technologies, special effects production usually requires relevant personnel to first produce three-dimensional model materials, and then render the materials to obtain 3D special effects. However, this method relies on the ability to manually produce 3D special effects, and some 3D special effects may be difficult to produce, with high costs and low efficiency. Summary of the invention

[0003] The embodiments of the present disclosure at least provide an image processing method, device, electronic device and storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides an image processing method, including:

[0005] Get the image to be processed;

[0006] Performing initial special effects processing on the image to be processed to obtain an initial special effects image;

[0007] Based on the target generation model, the initial special effects image is taken as input, target image information of the initial special effects image is extracted, and based on the target style special effects information, a target special effects image is generated whose semantics conform to preset conditions with the target image information, wherein the target generation model is used to generate a target style image.

[0008] In a second aspect, the present disclosure also provides an image processing device, including:

[0009] An acquisition module, used for acquiring an image to be processed;

[0010] A first processing module, used for performing initial special effect processing on the image to be processed to obtain an initial special effect image;

[0011] The second processing module is used to extract target image information of the initial special effects image based on the target generation model and the initial special effects image as input, and generate a target special effects image whose semantics conform to preset conditions with the target image information based on the target style special effects information, wherein the target generation model is used to generate a target style image.

[0012] In a third aspect, an optional implementation of the present disclosure further provides an electronic device, comprising a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the above-mentioned first aspect, or the steps in any possible implementation of the first aspect.

[0013] In a fourth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned first aspect, or the steps in any possible implementation of the first aspect.

[0014] In the disclosed embodiment, an image to be processed is obtained; initial special effects processing is performed on the image to be processed to obtain an initial special effects image; based on a target generation model, the initial special effects image is used as input to extract target image information of the initial special effects image, and based on the target style special effects information, a target special effects image is generated that semantically meets preset conditions with the target image information, wherein the target generation model is used to generate a target style image. In this way, for the image to be processed, initial special effects processing can be performed first, and then a target special effects image is generated based on the target generation model, thereby improving the efficiency of special effects generation and not relying on manual capabilities. Therefore, richer and more diverse special effects can be generated, and the rendering effect of special effects is improved. Furthermore, since the processing is not directly based on the captured real scene image, the requirements for the target generation model are also reduced, and the stability of the target generation model is improved, thereby improving the generation stability of the target special effects image.

[0015] For a description of the effects of the above-mentioned image processing device, electronic device, and computer-readable storage medium, please refer to the description of the above-mentioned image processing method, which will not be repeated here.

[0016] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure.

[0017] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown;

[0020] Figure 2 A schematic diagram showing the principle of a target generation model provided by an embodiment of the present disclosure is shown;

[0021] Figure 3 A flowchart showing a training process corresponding to the image processing method provided in an embodiment of the present disclosure;

[0022] Figure 4 The figure shows the principle effect diagram of the image generation model provided by the embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of an image processing device provided by an embodiment of the present disclosure is shown;

[0024] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0025] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0026] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0027] Research has found that the special effects tools provided in short video platforms, such as 3D special effects, usually require relevant personnel to first produce three-dimensional model materials, and then render the materials to obtain 3D special effects. However, this method relies on the ability to manually produce 3D special effects. The generated 3D special effects are limited and may not meet the actual needs of users. They are also costly and inefficient.

[0028] Based on the above research, the present disclosure provides an image processing method, which first performs initial special effects processing on the image to be processed to obtain an initial special effects image, and then generates a target special effects image with target style special effects information based on a target generation model. In this way, when using special effects, initial special effects processing can be performed first, and then the target special effects image can be generated based on the target generation model, which improves the efficiency of special effects generation and does not rely on manual capabilities. Therefore, richer and more diverse special effects can be generated. Moreover, since the processing is not directly based on the captured real scene image, the requirements for the target generation model are also reduced, and the stability of the target generation model is improved, thereby improving the generation stability and effect of the target special effects image.

[0029] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0030] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0031] To facilitate understanding of this embodiment, an image processing method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the image processing method provided in the embodiment of the present disclosure is generally an electronic device with certain computing capabilities, and the electronic device includes, for example: a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a mobile device, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc., wherein a personal digital assistant is a handheld electronic device with some functions of an electronic computer, which can be used to manage personal information, browse the Internet, send and receive emails, etc. It is generally not equipped with a keyboard and can also be called a handheld computer. In some possible implementations, the image processing method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0032] The image processing method provided by the embodiment of the present disclosure is described below by taking the execution subject as a server as an example.

[0033] See also Figure 1 FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure, the method comprising:

[0034] S101: Acquire an image to be processed.

[0035] The disclosed embodiments can be applied to short video shooting scenarios. When shooting short videos, users can choose to use special effects, such as 3D special effects, artificial intelligence (AI) special effects, etc., and the captured video frame images can be combined with the special effects in real time, and each frame of the target special effects image after the combination can be displayed. Specifically, for example, in a short video shooting scenario, executing step S101 may include: responding to a short video shooting instruction, obtaining an image to be processed of the current frame captured according to a preset frame interval.

[0036] Among them, for short video shooting scenes, it is necessary to display the effects of special effects in real time. In order to enhance user experience, improve computing efficiency, and avoid problems such as freezes or unsmoothness, in the embodiments of the present disclosure, subsequent special effects combined processing can be performed on each frame of the image to be processed, or the image to be processed can be obtained according to a preset frame interval. This is not limited in the embodiments of the present disclosure.

[0037] In addition, in the embodiment of the present disclosure, the acquired image to be processed may also be an image taken in advance, which is not limited in the embodiment of the present disclosure.

[0038] S102: Performing initial special effect processing on the image to be processed to obtain an initial special effect image.

[0039] Specifically for step S102, the present disclosure provides a possible implementation method, including: performing initial special effects processing on the image to be processed according to a preset image processing method to obtain an initial special effects image; wherein the preset image processing method includes at least one of the following: deformation processing, beauty and makeup processing, and filter processing.

[0040] For example, if the selected special effect is a 3D face cartoon special effect, face recognition can be performed on the image to be processed, and the recognized face can be deformed, beautified, and made up.

[0041] In the disclosed embodiment, the image to be processed is subjected to initial special effects processing, and the image processing method can be pre-set. For example, the image processing method can be specifically integrated into the basic special effects package tool, and then in actual application, the image to be processed is input into the basic special effects package tool to obtain the initial special effects image. Among them, the basic special effects package tool refers to a creation tool made based on a preset special effects open platform, etc. Usually, the capabilities provided by the special effects open platform are limited, and can only provide some basic capabilities, such as 2D deformation, 3D deformation, basic beauty, etc. Relevant personnel can design a general special effects shape and framework according to needs, such as 3D cartoon special effects, 3D face special effects, etc., then the relevant personnel can design a required 3D face model based on the special effects open platform, such as how long the face is, how big the ears are, etc., and then finally obtain the designed basic special effects package tool, so that all images can obtain an initial special effects image that meets the special effects capabilities in the basic special effects package tool after passing through the basic special effects package tool.

[0042] In this way, in the embodiment of the present disclosure, a basic special effects package tool can be pre-produced, and then initial special effects processing can be performed on the processed image based on the basic special effects package tool, which can improve the efficiency of the initial special effects processing and quickly generate the initial special effects image. In addition, relevant personnel can customize the initial special effects according to their needs, which is more flexible and can meet the needs of different users.

[0043] S103: Based on the target generation model, the initial special effects image is used as input to extract target image information of the initial special effects image, and based on the target style special effects information, a target special effects image is generated whose semantics meet preset conditions of the target image information, wherein the target generation model is used to generate a target style image.

[0044] In the disclosed embodiment, the target generation model is mainly used to further perform special effects processing. However, it is not actually desired to completely change the special effects of the basic special effects package tool after processing by the target generation model, because the basic special effects package tool is a special effects effect initially designed by relevant personnel, which is a rough prototype effect. The target generation model is used to further optimize the special effects, such as rendering some textures with different textures, etc., rather than completely changing the special effects. Therefore, in the disclosed embodiment, the target generation model will retain the approximate shape or outline of the initial special effects image when generating the target special effects image, and will process based on the target style special effects information, and ultimately generate a target special effects image that has the same approximate shape as the initial special effects image and is biased towards the target style.

[0045] Specifically for step S103, the present disclosure also provides a possible implementation method, including: based on the target generation model, taking the initial special effects image as input, performing feature extraction on the initial special effects image, and obtaining target image information of the initial special effects image, wherein the target image information at least includes image deformation information; based on the target style material information in the target style special effects information, processing the material in the initial special effects image, and generating a target special effects image that meets preset conditions with the image deformation information and is consistent with the target style material information.

[0046] Among them, the preset conditions, for example, are less than or equal to a preset similarity threshold, etc., which are not limited in the specific embodiments of the present disclosure. The purpose is to ensure that the target special effect image and the target image information in the initial special effect image are the same or substantially the same.

[0047] In the embodiments of the present disclosure, during the rendering process, the material can represent the combination of various visual attributes of the object surface, such as surface color, texture, smoothness, transparency, reflectivity, refractive index, luminosity, etc. These materials will basically not affect the main shape of the object. Therefore, in the embodiments of the present disclosure, the target generation model can retain the image deformation information of the initial special effects image, and based on the target style special effects information, transform the material of the initial special effects image, so as to generate the target special effects image.

[0048] In the disclosed embodiment, an image to be processed is obtained; initial special effects processing is performed on the image to be processed to obtain an initial special effects image; and then based on a target generation model, the initial special effects image is used as input to extract target image information of the initial special effects image; and based on the target style special effects information, a target special effects image is generated that meets preset conditions with the semantics of the target image information. In this way, initial special effects processing can be performed first, and then based on the target generation model, a target special effects image is generated, which improves the efficiency of special effects generation. There is no need to rely on manual capabilities, and the target generation model can be used to perform further special effects optimization to improve the richness and vividness of the special effects. Moreover, since the processing is not directly based on the captured real scene image, but the initial special effects image that has undergone certain initial special effects processing is used as input into the target generation model, the requirements for the target generation model are also reduced, and the stability of the target generation model is improved, thereby improving the generation stability and effect of the target special effects image.

[0049] In the embodiment of the present disclosure, the target generation model is, for example, a picture-generated picture model, which can be pre-trained so that the target generation model can convert the input image into a target style image. For the training method of the target generation model, the present disclosure also provides possible implementation methods, specifically including:

[0050] S1. Obtain a sample image pair set, wherein the sample image pair set includes a plurality of sample image pairs, and each sample image pair includes a first sample image and a corresponding second sample image.

[0051] In the disclosed embodiment, a set of sample image pairs of a training target generation model is used to train the target generation model to learn to generate target style images. Therefore, it can be understood that the first sample image in a sample image pair is an image to be converted, and the corresponding second sample image can be considered as a training label of the first sample image, that is, an image with a target style. Among them, obtaining the sample image pair set can be based on the image generation model and controlled by the target style text to generate images with the target style, thereby improving the generation efficiency of the sample image pair set.

[0052] S2. Input the sample image pair into the target generation model, reconstruct and predict the first sample image based on the generator in the target generation model to obtain a predicted image, and perform similarity judgment on the predicted image and the second sample image based on the discriminator in the target generation model to train the target generation model until a preset number of iterations is reached or the target loss function converges, wherein the target loss function at least includes a loss function between the second sample image and the predicted image.

[0053] The target generation model may be, for example, a generative adversarial network model, which is not limited in the embodiments of the present disclosure. Taking the target generation model as a generative adversarial network model as an example, the target generation model includes a generator and a discriminator. For example, see Figure 2 As shown, it is a schematic diagram of the principle of the target generation model in the embodiment of the present disclosure. The generator is mainly used to generate images, and its input is the first sample image. The output prediction image needs to imitate and learn the real second sample image as much as possible. The input of the discriminator is the real second sample image or the prediction image output by the generator. The main function is to be able to distinguish between true and false sample images, and to distinguish the prediction image output by the generator from the real second sample image as much as possible, so that the generator and the discriminator compete with each other and continuously adjust the network parameters. The ultimate goal is to obtain a higher-quality generator and a discriminator with stronger judgment ability. Both have good performance and meet the convergence of the target loss function or reach a preset number of iterations.

[0054] Among them, for the method of obtaining a sample image pair set for training a target generation model, the present disclosure also provides a possible implementation method, specifically obtaining a sample image pair set, including:

[0055] 1) Obtain an initial sample image set, wherein the initial sample image set includes a plurality of initial sample images.

[0056] 2) Performing initial special effects processing on the initial sample image to obtain a first sample image.

[0057] In the disclosed embodiment, relevant designers can customize and design personalized special effects based on the basic capabilities provided by the special effects open platform. The relevant image processing for producing the special effects can be packaged to obtain a basic special effects package tool, and then all initial sample images can be subsequently input into the basic special effects package tool for initial special effects processing, and images processed with corresponding special effects can be obtained. The initial special effects processing is more efficient and the batch processing speed is improved. In the disclosed embodiment, the image processed by the basic special effects package tool can be used as the first sample image in the training sample image pair.

[0058] 3) Based on the image generation model, a second sample image corresponding to the first sample image is generated according to the first sample image and the text description information of the target image, wherein the text description information of the target image at least includes the target style description information.

[0059] In the embodiments of the present disclosure, taking into account that the basic capabilities provided by the current special effects open platform are limited, relevant personnel may not be able to design special effects that meet the target style based on the special effects open platform. For example, some advanced material rendering effects may not be achieved based on the special effects open platform. Therefore, in the embodiments of the present disclosure, based on the image generation model, the first sample image is optimized in the target style, so that a second sample image that meets the target style can be generated.

[0060] Specifically, for generating a second sample image corresponding to a first sample image based on an image generation model according to the first sample image and the text description information of the target image, the present disclosure also provides a specific implementation method, including: based on the image generation model, taking the first sample image and the text description information of the target image as input, performing image feature extraction on the first sample image to obtain first image feature information of the first sample image, and performing noise processing on the first sample image according to the first image feature information and noise control parameters to obtain a noisy image vector representation; obtaining a text vector representation of the text description information of the target image, and reconstructing the second sample image according to the text vector representation and the noise image vector representation.

[0061] To facilitate understanding, the image generation model is briefly introduced below. In the embodiment of the present disclosure, the image generation model may be, for example, a stable diffusion model. The stable diffusion model mainly includes a diffusion process and an inverse diffusion process. The diffusion process is based on the principle of gradually adding Gaussian noise to destroy the training data until it becomes Gaussian white noise. The inverse diffusion process is to gradually remove noise from a Gaussian white noise, and finally obtain a clean and noise-free image. That is, the training process of the stable diffusion model is to add any degree of noise to the image in the training set, and predict the noise by optimizing the neural network. Through training, the neural network can learn to predict the noise of the picture from a noisy image, thereby having the ability to remove noise.

[0062] In addition, the stable diffusion model is a diffusion model based on latent coding. It does not perform diffusion and inverse diffusion processes directly in the pixel space of the image, but instead introduces a variational autoencoder to encode the image into a lower-dimensional latent code, thereby realizing diffusion and inverse diffusion processes in the feature space, which greatly reduces the training and inference time of the model. In addition, the stable diffusion model can also vectorize text information based on the text encoder, and combine text information with the image generation process through the self-attention mechanism, so that the generated image content can be controlled by text.

[0063] Therefore, in the embodiments of the present disclosure, the image-generating related tasks can be implemented based on the stable diffusion model, and the target image text description information can be added to control the image generation effect. In addition, in order to enable the stable diffusion model to finally output the image, not only can the general contour deformation and other features of the input image be maintained, but also its material style can be changed at the same time, therefore, when implementing the image-generating related tasks in the embodiments of the present disclosure based on the stable diffusion model, it is considered that if the inverse diffusion model starts from a Gaussian white noise, a completely random image will be obtained after denoising, but if the inverse diffusion process starts from a noise image with an object contour, the image obtained after denoising will have a contour or shape similar to the noise image, so that the present disclosure In the embodiment, with respect to the image generation model, the noise level in the diffusion process of the image generation model is controlled. The diffusion process does not generate a completely white noise, but generates a noisy image with a certain amount of noise through noise control parameters. The noisy image is then denoised through the inverse diffusion process of the image generation model. In the inverse diffusion process, the target image text description information is also combined to control the target style of the second sample image that is finally generated, so that a second sample image that has roughly the same outline or deformation as the first sample image and conforms to the target style can be obtained. The target style is not limited and can be controlled through the target image text description information to generate a second sample image of the desired personalized target style.

[0064] In this way, in the embodiment of the present disclosure, based on the image generation model, the first sample image can be transformed to generate a second sample image, thereby obtaining a set of sample image pairs that can train the target generation model, thereby improving the efficiency of obtaining the sample image pair set, and thus based on the sample image pair set, the target generation model can be trained to generate images of the target style.

[0065] The following uses a specific application scenario to illustrate. Figure 3 As shown, it is a flowchart of the training process corresponding to the image processing method provided in the embodiment of the present disclosure.

[0066] 1) If Figure 3As shown, during the training process, relevant personnel can first design original special effects based on the special effects open platform, such as basic capabilities provided by the special effects open platform such as 2D deformation, 3D deformation, etc., to obtain basic special effects package tools, and then can batch process the initial sample images in the initial sample image set based on the basic special effects package tools to obtain the first sample image, wherein the initial sample image can be a real captured image without any restriction.

[0067] 2) Based on the image generation model, a second sample image corresponding to the first sample image is generated according to the first sample image and the text description information of the target image.

[0068] The target image text description information at least includes target style description information. For example, if the target image text description information is to generate an image of style A, the generation of an image of a corresponding style can be controlled according to the rendering effect that the final special effect is intended to achieve.

[0069] See also Figure 4 As shown, it is a principle effect diagram of the image generation model provided in the embodiment of the present disclosure, such as Figure 4 As shown in FIG, based on the diffusion process of the image generation model, the first sample image is subjected to noise processing, and then denoised based on the inverse diffusion process of the image generation model. In the denoising process, the generated image content is controlled in combination with the target image text description information, and finally a second sample image is generated that can retain the general outline or deformation features in the first sample image and conform to the target style, as shown in FIG. Figure 4 As shown, after the first sample image is processed by the image generation model, the 3D special effects of the second sample image are better, the general outline remains basically unchanged, and the rendering effect of materials such as skin is improved. For example, the cartoon performance effect of the rendered skin material is more obvious, which improves the special effects rendering effect.

[0070] 3) In the embodiment of the present disclosure, based on the image generation model, a set of sample image pairs for training the target generation model can be obtained, each sample image pair includes a first sample image and a corresponding second sample image, and then based on the set of sample image pairs, the target generation model is trained, and finally the target generation model can be trained so that it can generate images of the target style.

[0071] Furthermore, in the actual application process, in the embodiment of the present disclosure, initial special effects processing can be performed on the image to be processed based on the basic special effects package tool to obtain an initial special effects image, and then it is processed based on the target generation model to generate a target special effects image whose target image information semantics meets preset conditions and is consistent with the target style special effects information with the initial special effects image, and then the final target special effects image can be returned to the client for display.

[0072] In this way, in the embodiment of the present disclosure, based on the image generation model and text control, a sample image set that meets the requirements can be automatically generated, thereby improving the generation efficiency, and thus the target generation model can be trained based on the sample image set, thereby improving the training efficiency and model effect. Moreover, in the training of the target generation model, the contour or deformation of the learned second sample image is roughly the same as the first sample image, rather than being greatly different, thereby improving the stability and learning effect of the target generation model, and improving the speed of training the target generation model. Furthermore, in actual applications, the target special effects image can be generated based on the target generation model after being processed by a basic special effects toolkit. There is no need to rely on the capabilities of the artificially produced basic special effects package tools, and the target generation model can be used for further special effects optimization, thereby improving the special effects generation efficiency, as well as improving the richness and vividness of the special effects.

[0073] Among them, the target generation model is, for example, a generative adversarial network model, and the image generation model is, for example, a stable diffusion model. The image generation of the target generation model is more realistic, and the target generation model requires fewer computing resources, which is suitable for deployment on terminal devices. However, the target generation model lacks stability and diversity, while the image generation model has better image generation performance, can generate diverse images, and can also obtain good image effects based on text descriptions. However, the image generation model requires large computing resources and is not suitable for deployment on terminal devices. It is usually deployed on the server side. Therefore, in the embodiment of the present disclosure, in actual applications, the target generation model is used to generate target special effect images. In order to solve the problem of the target generation model, a sample image pair set of the training target generation model that meets the requirements can be efficiently obtained based on the image generation model, thereby improving the efficiency and diversity of obtaining the sample image pair set. The obtained sample image pair set is not a randomly different image, but an image pair with roughly the same deformation. Then, based on the sample image pair set, the target generation model is trained, which can improve the stability and diversity of the target generation model, thereby improving the image generation performance of the target generation model.

[0074] In addition, in the embodiments of the present disclosure, there is no restriction on the network structure of the image generation model and the target generation model. For example, a flow-based generation model, a variational autoencoder model, etc. may also be used, and other methods may also be used to generate a set of sample image pairs for the target generation model, for example, mining images and obtaining corresponding image pairs based on some image processing tools, etc., which is not restricted in the embodiments of the present disclosure.

[0075] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0076] Based on the same inventive concept, an image processing device corresponding to the image processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned image processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0077] Reference Figure 5 FIG. 1 is a schematic diagram of an image processing device provided by an embodiment of the present disclosure, the device comprising:

[0078] An acquisition module 51 is used to acquire an image to be processed;

[0079] A first processing module 52, configured to perform initial special effect processing on the image to be processed to obtain an initial special effect image;

[0080] The second processing module 53 is used to extract the target image information of the initial special effects image based on the target generation model and the initial special effects image as input, and generate a target special effects image that meets the preset conditions with the semantics of the target image information based on the target style special effects information, wherein the target generation model is used to generate a target style image.

[0081] In an optional implementation, a training module 54 is further included, and the target generation model is obtained by the training module 54 through the following steps:

[0082] Acquire a sample image pair set, wherein the sample image pair set includes a plurality of sample image pairs, and each of the sample image pairs includes a first sample image and a corresponding second sample image;

[0083] The sample image pair is input into the target generation model, and based on the generator in the target generation model, the first sample image is reconstructed and predicted to obtain a predicted image, and based on the discriminator in the target generation model, the predicted image and the second sample image are judged for similarity to train the target generation model until a preset number of iterations is reached or the target loss function converges, wherein the target loss function at least includes a loss function between the second sample image and the predicted image.

[0084] In an optional implementation, when acquiring the sample image pair set, the training module 54 is used to:

[0085] Acquire an initial sample image set, wherein the initial sample image set includes a plurality of initial sample images;

[0086] Performing initial special effects processing on the initial sample image to obtain the first sample image;

[0087] Based on the image generation model, the second sample image corresponding to the first sample image is generated according to the first sample image and the target image text description information, wherein the target image text description information at least includes the target style description information.

[0088] In an optional implementation, when the image generation model is based on the first sample image and the target image text description information to generate the second sample image corresponding to the first sample image, the training module 54 is used to:

[0089] Based on the image generation model, taking the first sample image and the target image text description information as input, performing image feature extraction on the first sample image to obtain first image feature information of the first sample image, and performing noise processing on the first sample image according to the first image feature information and noise control parameters to obtain a noise image vector representation;

[0090] A text vector representation of the target image text description information is obtained, and the second sample image is reconstructed based on the text vector representation and the noise image vector representation.

[0091] In an optional implementation, when the target-based generation model takes the initial special effect image as input, extracts the target image information of the initial special effect image, and generates a target special effect image that semantically meets the preset conditions with the target image information based on the target style special effect information, the training module 54 is used to:

[0092] Based on the target generation model, taking the initial special effect image as input, performing feature extraction on the initial special effect image to obtain target image information of the initial special effect image, wherein the target image information at least includes image deformation information;

[0093] Based on the target style material information in the target style special effect information, the material in the initial special effect image is processed to generate a target special effect image that meets preset conditions with the image deformation information and is consistent with the target style material information.

[0094] In an optional embodiment, when the image to be processed is subjected to initial special effects processing to obtain an initial special effects image, the first processing module 52 is used to: perform initial special effects processing on the image to be processed according to a preset image processing method to obtain an initial special effects image; wherein the preset image processing method includes at least one of the following: deformation processing, beauty and makeup processing, and filter processing.

[0095] In an optional implementation, when acquiring the image to be processed, the acquisition module 51 is used to: respond to a short video shooting instruction and acquire the image to be processed of the current frame shot according to a preset frame interval.

[0096] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0097] The present disclosure also provides an electronic device, such as Figure 6 FIG. 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, including:

[0098] A processor 61 and a memory 62; the memory 62 stores machine-readable instructions executable by the processor 61, and the processor 61 is used to execute the machine-readable instructions stored in the memory 62. When the machine-readable instructions are executed by the processor 61, the processor 61 is used to perform the following steps:

[0099] Get the image to be processed;

[0100] Performing initial special effects processing on the image to be processed to obtain an initial special effects image;

[0101] Based on the target generation model, the initial special effects image is taken as input, target image information of the initial special effects image is extracted, and based on the target style special effects information, a target special effects image is generated whose semantics conform to preset conditions with the target image information, wherein the target generation model is used to generate a target style image.

[0102] In an optional implementation manner, the target generation model is obtained by training the processor 61 through the following steps:

[0103] Acquire a sample image pair set, wherein the sample image pair set includes a plurality of sample image pairs, and each of the sample image pairs includes a first sample image and a corresponding second sample image;

[0104] The sample image pair is input into the target generation model, and based on the generator in the target generation model, the first sample image is reconstructed and predicted to obtain a predicted image, and based on the discriminator in the target generation model, the predicted image and the second sample image are judged for similarity to train the target generation model until a preset number of iterations is reached or the target loss function converges, wherein the target loss function at least includes a loss function between the second sample image and the predicted image.

[0105] In an optional implementation, when acquiring the sample image pair set, the processor 61 is used to:

[0106] Acquire an initial sample image set, wherein the initial sample image set includes a plurality of initial sample images;

[0107] Performing initial special effects processing on the initial sample image to obtain the first sample image;

[0108] Based on the image generation model, the second sample image corresponding to the first sample image is generated according to the first sample image and the target image text description information, wherein the target image text description information at least includes the target style description information.

[0109] In an optional implementation manner, when the second sample image corresponding to the first sample image is generated based on the image generation model according to the first sample image and the target image text description information, the processor 61 is configured to:

[0110] Based on the image generation model, taking the first sample image and the target image text description information as input, performing image feature extraction on the first sample image to obtain first image feature information of the first sample image, and performing noise processing on the first sample image according to the first image feature information and noise control parameters to obtain a noise image vector representation;

[0111] A text vector representation of the target image text description information is obtained, and the second sample image is reconstructed based on the text vector representation and the noise image vector representation.

[0112] In an optional implementation manner, when the target-based generation model takes the initial special effect image as input, extracts the target image information of the initial special effect image, and generates a target special effect image that semantically meets the preset conditions with the target image information based on the target style special effect information, the processor 61 is used to:

[0113] Based on the target generation model, taking the initial special effect image as input, performing feature extraction on the initial special effect image to obtain target image information of the initial special effect image, wherein the target image information at least includes image deformation information;

[0114] Based on the target style material information in the target style special effect information, the material in the initial special effect image is processed to generate a target special effect image that meets preset conditions with the image deformation information and is consistent with the target style material information.

[0115] In an optional embodiment, when the image to be processed is subjected to initial special effects processing to obtain an initial special effects image, the processor 61 is used to: perform special effects processing on the image to be processed according to a preset image processing method to obtain an initial special effects image; wherein the image processing method includes at least one of the following: deformation processing, beauty and makeup processing, and filter processing.

[0116] In an optional implementation, when acquiring the image to be processed, the processor 61 is used to: respond to a short video shooting instruction and acquire the image to be processed of the current frame shot according to a preset frame interval.

[0117] The above-mentioned memory 62 includes internal memory 621 and external memory 622; the memory 621 here is also called internal memory, which is used to temporarily store the calculation data in the processor 61, as well as the data exchanged with the external memory 622 such as the hard disk. The processor 61 exchanges data with the external memory 622 through the internal memory 621.

[0118] The specific execution process of the above instructions can refer to the steps of the image processing method described in the embodiment of the present disclosure, which will not be repeated here.

[0119] The present disclosure also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0120] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the image processing method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0121] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0122] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0123] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0125] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0126] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Get the image to be processed; Performing initial special effects processing on the image to be processed to obtain an initial special effects image; Based on the target generation model, the initial special effects image is taken as input, target image information of the initial special effects image is extracted, and based on the target style special effects information, a target special effects image is generated whose semantics conform to preset conditions with the target image information, wherein the target generation model is used to generate a target style image.

2. The method according to claim 1, characterized in that The target generation model is obtained by training through the following steps: Acquire a sample image pair set, wherein the sample image pair set includes a plurality of sample image pairs, and each of the sample image pairs includes a first sample image and a corresponding second sample image; The sample image pair is input into the target generation model, and based on the generator in the target generation model, the first sample image is reconstructed and predicted to obtain a predicted image, and based on the discriminator in the target generation model, the predicted image and the second sample image are judged for similarity to train the target generation model until a preset number of iterations is reached or the target loss function converges, wherein the target loss function at least includes a loss function between the second sample image and the predicted image.

3. The method according to claim 2, characterized in that The step of obtaining a set of sample image pairs comprises: Acquire an initial sample image set, wherein the initial sample image set includes a plurality of initial sample images; Performing initial special effects processing on the initial sample image to obtain the first sample image; Based on the image generation model, the second sample image corresponding to the first sample image is generated according to the first sample image and the target image text description information, wherein the target image text description information at least includes the target style description information.

4. The method according to claim 3, characterized in that The generating the second sample image corresponding to the first sample image based on the image generation model according to the first sample image and the target image text description information includes: Based on the image generation model, taking the first sample image and the target image text description information as input, performing image feature extraction on the first sample image to obtain first image feature information of the first sample image, and performing noise processing on the first sample image according to the first image feature information and noise control parameters to obtain a noise image vector representation; A text vector representation of the target image text description information is obtained, and the second sample image is reconstructed based on the text vector representation and the noise image vector representation.

5. The method according to any one of claims 1 to 4, characterized in that: The target-based generation model takes the initial special effect image as input, extracts target image information of the initial special effect image, and generates a target special effect image that semantically meets preset conditions of the target image information based on the target style special effect information, including: Based on the target generation model, taking the initial special effect image as input, performing feature extraction on the initial special effect image to obtain target image information of the initial special effect image, wherein the target image information at least includes image deformation information; Based on the target style material information in the target style special effect information, the material in the initial special effect image is processed to generate a target special effect image that meets preset conditions with the image deformation information and is consistent with the target style material information.

6. The method according to claim 1, characterized in that The performing initial special effects processing on the image to be processed to obtain an initial special effects image includes: According to a preset image processing method, performing initial special effects processing on the image to be processed to obtain an initial special effects image; Among them, the preset image processing method includes at least one of the following: deformation processing, beauty and makeup processing, and filter processing.

7. The method according to claim 1, characterized in that The step of obtaining the image to be processed comprises: In response to the short video shooting instruction, the image to be processed of the current frame shot is obtained according to the preset frame interval.

8. An image processing device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; A first processing module, used for performing initial special effect processing on the image to be processed to obtain an initial special effect image; The second processing module is used to extract target image information of the initial special effects image based on the target generation model and the initial special effects image as input, and generate a target special effects image whose semantics conform to preset conditions with the target image information based on the target style special effects information, wherein the target generation model is used to generate a target style image.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor performs the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images