Image processing method and device, electronic equipment and storage medium

Abstract

The method comprises: collecting a to-be-processed image; determining to-be-processed Gaussian noise corresponding to the to-be-processed image; processing the to-be-processed Gaussian noise based on a first feature fusion function corresponding to at least one convolution layer in a first style image generation model and a second feature fusion function of at least one convolution layer in a second style image generation model to obtain a target special effect image fused with first style features and second style features; the first style features correspond to the first style image generation model, and the second style features correspond to the second style image generation model. The technical solution provided in the embodiments of the present disclosure can efficiently generate images fused with two styles without a large number of training samples fused with the two styles, thereby reducing consumption in the model construction process and improving image processing effect.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of data processing, and particularly relate to an image processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the continuous development of image processing technology, users can use various applications to process images, so that the processed images present the style type they expect.

[0003] In the prior art, related algorithms for image processing often need to train a model using a large amount of data before providing corresponding services to users. However, this approach consumes a large amount of cost, for example, it is necessary to draw or use sample data of a certain style, which is costly. Furthermore, if multiple styles are superimposed, not only does it increase the difficulty of obtaining sample data, but also the sample quality is poor, and accordingly, the result of the model trained based on such samples is also less realistic. SUMMARY

[0004] Embodiments of the present disclosure provide an image processing method, device, electronic equipment and storage medium, which can generate a special effect image with two styles fused without a large number of training samples with two style types fused, thereby improving the convenience and efficiency of generating a special effect image.

[0005] In a first aspect, embodiments of the present disclosure provide an image processing method, which comprises:

[0006] acquiring a to-be-processed image;

[0007] determining to-be-processed Gaussian noise corresponding to the to-be-processed image, processing the to-be-processed Gaussian noise based on a first feature fusion function corresponding to at least one convolution layer in a first style image generation model and a second feature fusion function of at least one convolution layer in a second style image generation model, and obtaining a target special effect image with first style features and second style features fused;

[0008] wherein the first style features correspond to the first style image generation model, and the second style features correspond to the second style image generation model.

[0009] In a second aspect, embodiments of the present disclosure also provide an image processing method, which comprises:

[0010] acquiring a to-be-processed image and determining to-be-processed Gaussian noise corresponding to the to-be-processed image;

[0011] generate the target special effect image fused with the first style feature and the second style feature based on the third image generation model.

[0012] The third image generation model is trained based on training samples constructed based on the first feature fusion function of at least one convolution layer in the first style image generation model and the second feature fusion function of at least one convolution layer in the second style image generation model, the first style image generation model corresponds to the first style feature, and the second style image generation model corresponds to the second style feature.

[0013] In a third aspect, the embodiments of the present disclosure further provide an image processing device, which comprises:

[0014] an image acquisition module configured to acquire a to-be-processed image;

[0015] a special effect image generation model configured to determine to-be-processed Gaussian noise corresponding to the to-be-processed image, process the to-be-processed Gaussian noise based on a first feature fusion function corresponding to at least one convolution layer in a first style image generation model and a second feature fusion function of at least one convolution layer in a second style image generation model, and obtain a target special effect image fused with the first style feature and the second style feature.

[0016] In a fourth aspect, the embodiments of the present disclosure further provide an image processing device, which comprises:

[0017] a Gaussian noise processing module configured to acquire a to-be-processed image and determine to-be-processed Gaussian noise corresponding to the to-be-processed image;

[0018] a special effect image determination module configured to process the to-be-processed Gaussian noise based on a third image generation model and obtain a target special effect image fused with a first style feature and a second style feature.

[0019] The third image generation model is trained based on training samples constructed based on the first feature fusion function of at least one convolution layer in the first style image generation model and the second feature fusion function of at least one convolution layer in the second style image generation model, the first style image generation model corresponds to the first style feature, and the second style image generation model corresponds to the second style feature.

[0020] In a fifth aspect, the embodiments of the present disclosure further provide an electronic device, which comprises:

[0021] one or more processors;

[0022] a storage device configured to store one or more programs,

[0023] The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of image processing according to any of the embodiments of the present disclosure.

[0024] In a sixth aspect, the embodiments of the present disclosure further provide a storage medium containing computer executable instructions for executing the method of image processing according to any of the embodiments of the present disclosure when executed by a computer processor.

[0025] The technical scheme of the embodiments of the present disclosure, the first style image generation model and the second style image generation model are trained based on the model parameter migration of the face image generation model, therefore, the corresponding model can be trained under the condition of limited amount of samples. Further, when the user triggers the special effect display operation is detected, the collected image to be processed can be processed to obtain Gaussian noise, and further, based on the first style image generation model, the second style image generation model and at least one convolution layer deployed on the terminal device, the corresponding feature fusion function is used to process the Gaussian noise to be processed, to obtain the target special effect image fused with the first style feature and the second style feature, which solves the problem that in the prior art, it is difficult to obtain sample images fused with the first style feature and the second style feature, i.e., the cost of obtaining samples is high, and in the case of poor sample quality, the image output by the special effect model trained is also poor in effect, which realizes the construction of the corresponding style model by using the transfer learning method, so that the first style image generation model and the second style image generation model can be trained with a small amount of style image samples, and further, based on the first style image generation model and the second style image generation model, the special effect image fused with the corresponding style feature can be obtained when the image is processed. BRIEF DESCRIPTION OF DRAWINGS

[0026] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals can refer to the same or similar elements. It should be understood that the drawings are schematic, and the original and elements are not necessarily drawn according to the scale.

[0027] Figure 1 A flowchart of a process for training a face image generation model according to an embodiment of the present disclosure is shown in FIG. 1.

[0028] Figure 2 A structural diagram of a face image generation model (stylegan model) according to an embodiment of the present disclosure is shown in FIG. 2.

[0029] Figure 3 A flowchart of an image processing method according to an embodiment of the present disclosure is shown in FIG. 3.

[0030] Figure 4 A schematic diagram of processing a to-be-processed image based on a first style image generation model and a second style image generation model according to an embodiment of the present disclosure is shown in FIG. 1;

[0031] Figure 5 A flowchart of a method of image processing according to an embodiment of the present disclosure is shown in FIG. 2;

[0032] Figure 6 A schematic diagram of processing based on a feature fusion function corresponding to each convolutional layer according to an embodiment of the present disclosure is shown in FIG. 3;

[0033] Figure 7 A flowchart of a method of image processing according to an embodiment of the present disclosure is shown in FIG. 4;

[0034] Figure 8 A flowchart of a method of image processing according to an embodiment of the present disclosure is shown in FIG. 5;

[0035] Figure 9a A structural schematic diagram of processing a to-be-processed image based on a third image generation model and a face image generation model according to an embodiment of the present disclosure is shown in FIG. 6;

[0036] Figure 9b A structural schematic diagram of processing a to-be-processed image based on a third image generation model and a face image generation model according to an embodiment of the present disclosure is shown in FIG. 7;

[0037] Figure 10 A structural block diagram of an image processing device according to an embodiment of the present disclosure is shown in FIG. 8;

[0038] Figure 11 A structural block diagram of an image processing device according to an embodiment of the present disclosure is shown in FIG. 9;

[0039] Figure 12 A structural schematic diagram of an electronic device according to an embodiment of the present disclosure is shown in FIG. 10. DETAILED DESCRIPTION

[0040] Embodiments of the present disclosure will be described in more detail by referring to the drawings. Although certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms, and should not be construed as being limited to the embodiments set forth herein, but rather, these embodiments are provided so as to more completely and thoroughly understand the present disclosure. It is understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the scope of protection of the present disclosure.

[0041] It should be understood that each of the steps recited in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit performing the steps shown. The scope of the present disclosure is not limited in this regard.

[0042] The term "comprises" and variations thereof used in the present disclosure are open-ended, that is, "comprising but not limited to". The term "based on" is "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related definitions of other terms will be given in the description below.

[0043] It should be noted that the "first", "second", and the like concepts mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0044] It should be noted that the modification of "one" or "multiple" mentioned in the present disclosure is illustrative and not limiting, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0045] It can be understood that before using the technical solutions disclosed by the embodiments of the present disclosure, the type, scope of use, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0046] For example, in response to receiving the active request of the user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using the personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as electronic devices, application programs, servers or storage media that perform the operation of the technical solutions of the present disclosure according to the prompt information.

[0047] As an optional but non-limiting implementation, in response to receiving the active request of the user, the way of sending prompt information to the user may, for example, be the way of a pop-up window, in which the prompt information can be presented in the form of text. In addition, the pop-up window can also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0048] It can be understood that the above notification and user authorization process is only illustrative and does not limit the implementation of the present disclosure, and other ways that meet the relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0049] It can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws and regulations and relevant provisions.

[0050] Before introducing the technical solution, the application scenario can be exemplarily described. The technical solution of the present disclosure can be applied to any picture that needs special effect display, for example, it can be applied to the video shooting process, the image corresponding to the user being shot can be displayed with special effects, such as in the short video shooting scene, it can also be integrated into any image shooting scene, for example, in the camera with shooting function in the system, so that after shooting the image, the target special effect image fused with two style features can be obtained.

[0051] It should be noted that although the existing technology can realize the fusion of images with two style features, a large number of samples of images with two style features are needed, and then the model is trained based on a large number of samples. The above-mentioned method not only has the problems of inconvenient sample determination and uneven sample quality, but also has the problem of complex model training, which leads to the problems of poor processing effect of the trained model and difficult training, and accordingly, the special effect image obtained also has large difference from the actual one, resulting in poor user experience.

[0052] Figure 1 A flowchart of training a face image generation model is provided in the embodiment of the present disclosure. The embodiment can be applied to any scene that needs to train a face image generation model. The method can be executed by an image processing device, which can be realized in the form of software and / or hardware. The hardware can be an electronic device, such as a mobile terminal, a PC terminal or a server, etc. The scene of any image display is usually realized by cooperation of a client and a server. The method provided in the embodiment can be executed by the server, the client or the cooperation of the client and the server.

[0053] It should be noted that the first style image generation model and the second style image generation model are both models for generating images of different style types. The model is mainly constructed based on a pre-trained face image generation model and a target compilation model, i.e. the model parameters of the face image generation model are migrated to construct the model, and a small amount of style feature images are used for training. Based on this, the determination of the face image generation model can be introduced first. The face image generation model is mainly trained based on randomly sampled Gaussian noise, and at the same time, a generative adversarial method is used for training.

[0054] In actual application, the face image generation model can be a stylegan model based on a generative adversarial network. The generative adversarial network is composed of a generation network and a discrimination network. The generation network randomly samples from a latent space as input, and the output result needs to imitate the real samples in the training set as much as possible. The input of the discrimination network is the real sample and the output of the generation network. Based on this, it can be understood that the stylegan model in the embodiment also includes a generator and a discriminator. Specifically, the generator can be used to process the Gaussian noise corresponding to the user's face image, thereby regenerating a user's face image; the discriminator can be used to adjust the related parameters in the generator. The advantage of using the discriminator containing the discrimination network is that the user's face image regenerated by the stylegan model after parameter correction is almost completely consistent with the user's face image corresponding to the Gaussian noise as input. It should be noted that in the field of high-definition image generation, the stylegan model has very excellent expression ability and can at least generate high-definition pictures with a resolution of .

[0055] Further, the structural diagram in the stylegan model can be referred to as Figure 2 . The model includes at least one pooling layer and at least one convolutional layer. Each convolutional layer can further process the features processed by the pooling layer, and the processing dimension presents an increasing trend according to the input-output relationship. Optionally, the convolutional layer of the first level processes features with a resolution of , the convolutional layer of the second level processes features with a resolution of , and the convolutional layer of the last level processes features with a resolution of . The structure in each convolutional layer can refer to the existing stylegan model.

[0056] As Figure 1 indicated, the method includes:

[0057] S110, processing the Gaussian noise based on the to-be-trained image generator to obtain a to-be-discriminated image.

[0058] The model parameters in the to-be-trained image generator are initial model parameters. A plurality of training samples can be used to train the to-be-trained image generator to obtain the face image generation model. The training samples correspond to Gaussian noise including target subject face information, wherein the target subject face information is an image containing user face information, for example, a user's ID photo or a life photo, and the Gaussian noise can be understood as a high-dimensional vector corresponding to the target subject face information. It should be noted that in actual application, a large number of Gaussian noises can be obtained based on a large public data set FFHQ (the data set is a face feature data set).

[0059] It should be noted that the way of generating the face image generation model based on each Gaussian noise is the same, and an example of processing based on one of the Gaussian noises is described here.

[0060] Specifically, after the Gaussian noise is input into the to-be-trained face generator, the to-be-trained face generator can process the Gaussian noise to obtain a to-be-judged image corresponding to the Gaussian noise.

[0061] S120, based on the to-be-trained discriminator, the to-be-judged image and the real face image are discriminated to obtain a reference loss value, and the model parameters in the to-be-trained image generator are corrected based on the reference loss value.

[0062] According to the above description, it can also be determined that when the to-be-trained face image generation model is a stylegan model, the model is composed of a to-be-trained image generator and a discriminator.

[0063] A large number of Gaussian noises are processed by the to-be-trained image generator to generate to-be-judged images, i.e. images that may differ from the real face image input by the user. Further, the loss value between the to-be-judged image and the real face image can be determined based on the discriminator. That is, the discriminator needs to judge whether the image output by the to-be-trained face generator is real. Based on the real image and the to-be-judged image, a reference loss value can be determined. The model parameters in the to-be-trained face image generator and the to-be-trained discriminator can be corrected based on the reference loss value.

[0064] S130, converging the loss function in the to-be-trained image generator and the to-be-trained discriminator as a training target to obtain a face image generation model.

[0065] It can be understood that when the reference loss value is used to correct the model parameters in the to-be-trained image generation model, the training error of the loss function in the to-be-trained image generator and the to-be-trained discriminator, i.e. the loss parameter, can be used as a condition for detecting whether the loss function converges, such as whether the training error is less than a preset error or whether the error change trend is stable, or whether the current iteration number is equal to a preset number. If the convergence condition is detected, such as the training error of the loss function is less than the preset error, or the error change trend is stable, it indicates that the to-be-trained image generation model is trained, and at this time the iteration training can be stopped. If it is detected that the current convergence condition is not reached, other basic training samples can be further obtained to continue training the model until the training error of the loss function is within a preset range. It can be understood that when the training error of the loss function converges, the trained face image generation model is obtained, and at this time the Gaussian vector corresponding to the user's face image is input into the model to obtain an image that is almost completely consistent with the user's face image.

[0066] In this embodiment, the face image generation model can be a neural network model for generating a user face image. It can be understood that, after inputting a user face-related feature into the face image generation model, a face image consistent with the user face feature can be obtained after model processing.

[0067] The technical solution provided by the embodiments of the present disclosure can train a corresponding face image generation model based on Gaussian noise, achieving the effect of convenient training of the face image generation model, and then outputting a corresponding face image based on the face image generation model. Since Gaussian noise is relatively easy to obtain, training the model based on this can improve the convenience of model training. Furthermore, by migrating the model parameters in the face image generation model, a corresponding style image generation model can be trained under the condition of having a small number of style feature samples, improving the accuracy and convenience of subsequent model training.

[0068] Based on the above technical solution, it can be known that the input parameter of the face image generation model is Gaussian noise, and the output parameter is a corresponding face image. However, in actual application, the input is mostly a face image. Based on this, a to-be-trained compilation model can be reconstructed and trained to obtain a target compilation model, so as to convert the input face image into corresponding Gaussian noise based on the target compilation model, and then input the Gaussian noise into the face image generation model to output a corresponding face image.

[0069] That is, when deploying the model to a terminal device, since the terminal device generally has the function of collecting a user face image, and the trained face image generation model can only process Gaussian noise corresponding to the user face image. Therefore, in order to enable the special effect image generation model to run effectively on the terminal device, a model capable of generating corresponding Gaussian noise based on the user face image, i.e., a target compilation model, needs to be determined.

[0070] Among them, the to-be-trained compilation model can be an encoder model. Those skilled in the art should understand that the encoder-decoder framework is a deep learning type of model framework, and the embodiments of the present disclosure will not be repeated here.

[0071] The training process of the specific to-be-trained compilation model is as follows: a plurality of training images are obtained; for each to-be-trained image, the current training image is input into the to-be-trained compilation model to obtain to-be-used Gaussian noise corresponding to the current training image; the to-be-used Gaussian noise is input into the face image generation model to obtain an actual output image; based on the actual output image and the current training image, an image loss value is determined; based on the image loss value, the model parameters in the to-be-trained compilation model are corrected, and convergence of a loss function in the to-be-trained compilation model is taken as a training target to obtain a target compilation model.

[0072] In this embodiment, after obtaining the training images containing the facial features of the user, the to-be-trained compilation model can be used to process a plurality of these images to generate corresponding to-be-used Gaussian noise, which is actually a high-dimensional vector that cannot accurately and completely reflect the facial features of the user. Further, the face image generation model can be used to process these to-be-used Gaussian noise to obtain actual output images that are not completely consistent with the training images. After the actual output images and the current training image are determined, a plurality of loss values corresponding to the discriminator can be determined. When the plurality of loss values are used to correct the model parameters in the to-be-trained compilation model, the training error of the loss function in the model, i.e., the loss parameter, can be used as a condition for detecting whether the loss function converges, such as whether the training error is less than a preset error or whether the error change trend is stable, or whether the current iteration number is equal to a preset number. If it is detected that the convergence condition is reached, such as the training error of the loss function is less than the preset error or the error change trend is stable, it indicates that the training of the to-be-trained compilation model is completed, and at this time, the iteration training can be stopped. If it is detected that the current convergence condition is not reached, other training images can be further processed, and actual output images corresponding to the obtained Gaussian vectors can be generated based on the face image generation model, and the model can be continuously trained until the training error of the loss function is within a preset range. When the training error of the loss function converges, the target compilation model trained is obtained. It can be understood that the target compilation model is used to process an input face image into corresponding Gaussian noise, and after the face image of the user is input into the target compilation model, the face image generation model can output an image that is almost completely consistent with the face image of the user based on the Gaussian noise output by the target compilation model.

[0073] The technical solution provided in the embodiments of the present disclosure can train a target compilation model after training the face image generation model, so that the corresponding model can be deployed on a terminal device. The target compilation model can be used to process an input image to obtain corresponding Gaussian noise, and then the face image generation model can be used to process the Gaussian noise to output a corresponding face image, thereby achieving the effect that the trained model can be deployed on a terminal device for use.

[0074] Figure 3 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown in the foregoing embodiment. On the basis of the foregoing embodiment, a to-be-trained style image generation model can be constructed based on a pre-trained face image generation model and a target compilation model. Then, the to-be-trained style image generation model is trained based on data of corresponding style features, to obtain a first style image generation model and a second style image generation model.

[0075] It should be noted that it is difficult to obtain training samples, i.e., sample images, under a certain style feature, and the quality of the obtained training samples is uneven. Therefore, it is difficult to train a model of a corresponding style feature based on a certain style sample, or the image generation effect of the trained style image generation model is poor. In order to improve the accuracy and low cost of the style image generation model, after the face image generation model is trained, the model parameters in the face image generation model can be used to construct a corresponding to-be-trained model in a manner of transfer learning. Then, the first style image generation model and the second style image generation model can be trained based on a small amount of sample data of style features. The first style image generation model and the second style image generation model use training samples of different style types, and correspondingly, the output images correspond to different style features. The first style image generation model can correspond to a first style feature, and the second style image generation model can correspond to a second style feature.

[0076] Next, how to train the first style image generation model and the second style image generation model is described in detail.

[0077] In S310, model parameters in the face image generation model are obtained, and a first model and a second model are constructed based on the model parameters.

[0078] Generally, training a model to generate images of a specific style requires acquiring a large number of images belonging to that style as training samples. However, samples of a particular style are almost nonexistent or difficult to obtain. Consequently, in practical applications, it is impossible to train a model for this style, and therefore, it is impossible to convert captured images into images of this style. Therefore, in this embodiment, after training the parameters of the facial image generation model, transfer learning can be used to obtain a model for generating images of a specific style. In the field of artificial intelligence, transfer learning aims to apply knowledge or patterns learned in one domain or task to different but related domains or problems, i.e., to transfer labeled data or knowledge structures from related domains to improve or enhance the learning effect of the target domain or task. In this embodiment, the advantage of using transfer learning is that a model for generating a specific style can be trained with only a small number of samples.

[0079] It is understandable that the advantage of using transfer learning to build the first and second models is that it can efficiently build a model for generating images of a specific style using the already trained model parameters. This not only avoids the tedious process of obtaining a large number of images of that style as training data, thus eliminating the problem of difficulty in obtaining samples, but also reduces the consumption of computing resources.

[0080] by Figure 4 Taking an example, once the facial image generation model is determined to be G1, the model parameters of G1 can be obtained, and a second model G2 and a third model G3 can be generated based on transfer learning. Figure 4 It can be seen that after inputting the Gaussian noise corresponding to the user's facial image into G2 for processing, the image output by the model retains the user's unique facial features while presenting a stylized image. For example, the image under the first style feature output by G2 can be an image with added features such as clothing, hairstyle, hair ornaments, and makeup characteristic of the era on the basis of the user's original facial features. After inputting the Gaussian noise corresponding to the user's facial image into G3 for processing, the image output by the model retains the user's unique facial features while presenting an ancient style. For example, the image under the second style type output by G3 can be an image with added features of figures in ancient paintings on the basis of the user's original facial features. This can be understood as making the user's realistic facial image present the visual effect of ancient figure paintings.

[0081] S320. The first model to be trained is trained based on the training samples in the first style feature sample set to obtain the first style image generation model.

[0082] The first style feature can be a time style feature image, for example, a face image of a certain time style dress. The style dress corresponds to a certain time period. It can be understood that the first style feature is a style feature that presents the characteristics of a certain time user's clothing, hairstyle, hair accessories, and makeup. The image in the training sample corresponding to the first style feature, for example, can be a style feature of a certain dynasty, or a style feature of a certain period, that is, the first style feature corresponds to a time period. The image in the training sample can be processed based on the trained target compilation model to generate Gaussian noise corresponding to the face image. Figure 4 For example, when the first to-be-trained model is a model for generating a specific time feature image, the corresponding training sample is a plurality of images of the user's dress style of that time.

[0083] The process of training the first to-be-trained model is to obtain a plurality of training samples under the first style feature; input the Gaussian noise into the first to-be-trained sample generation model to obtain a first actual output image; based on the discriminator, the first actual output image and the corresponding first face image are discriminated to determine the loss value, and the model parameters in the first to-be-trained sample generation model are corrected based on the loss value; the loss function in the first to-be-trained image generation model is converged as a training target to obtain a first style image generation model based on the above image model training to obtain a first style image generation model.

[0084] It should be noted that since the first to-be-trained sample generation model is constructed based on the trained face image generation model, only a small amount of training samples of the first style feature is required to train the model, that is, the first target sample generation model can be obtained. In actual application, the training sample can be about 200 images of the first style feature, and at the same time, these images should have similar structures as the face image input by the user, for example, the image should have the user's facial features and hair features.

[0085] In this way, not only the convenience of model training is improved, but also the corresponding target sample generation model can be trained when the number of images of a specific style feature is small, greatly reducing the demand for training samples of the to-be-trained model.

[0086] S330, training the second to-be-trained model based on the training sample in the second style feature sample set to obtain a second style image generation model.

[0087] The second style feature is an ancient style material image, for example, an image in the style of ancient figure painting. It can be understood that the second style feature is a style feature that presents the characteristics of ancient gongbi painting and oil painting. Each training sample includes a second face image under the second style feature. After processing the second face image, Gaussian noise reflecting the corresponding face feature can be obtained, and the second face image can be generated by adding Gaussian noise to the first face image. Figure 4 For example, when the second to-be-trained sample generation model is a model for generating ancient style material style images, the corresponding training sample is a plurality of ancient style material style images.

[0088] The process of training the second to-be-trained sample generation model is the same as the training process of the first to-be-trained sample generation model. The only difference is the training sample used. The specific training process can be referred to the detailed description above.

[0089] The technical solution provided by the embodiments of the present disclosure can use the transfer learning method to construct the first to-be-trained model and the second to-be-trained model, and then obtain the first image generation model and the second image generation model under the condition of a small amount of style feature data, thereby improving the accuracy and convenience of obtaining the image generation model. Further, based on the above two models, a target special effect image that fuses the first style feature and the second style feature can be obtained, thereby improving the convenience and realism of determining the special effect image.

[0090] Figure 5 For the image processing method flowchart provided by the embodiments of the present disclosure, on the basis of the foregoing embodiments, the first style image generation model and the second style image generation model trained can be deployed on a terminal device, so that when an image to be processed is input, the first style image generation model and the second style image generation model can be used for fusion processing to obtain a target special effect image that fuses the first style feature and the second style feature. The specific implementation can be referred to the detailed description of the technical solution. The same or corresponding technical terms as the above embodiments are not repeated here.

[0091] As shown in Figure 5 the method comprises:

[0092] S510, collecting an image to be processed.

[0093] The device for performing the special effect image processing method provided in the embodiments of the present disclosure can be integrated in application software supporting special effect image processing functions, and the software can be installed in an electronic device. Optionally, the electronic device can be a mobile terminal or a PC terminal, etc. The application software can be a type of software for image / video processing, and specific application software will not be described here, as long as it can realize image / video processing. It can also be a specially developed application program to realize the software of adding special effects and special effect display, or be integrated in the corresponding page, and the user can realize special effect adding processing through the integrated page in the PC terminal.

[0094] In the embodiments, in the application software or application program supporting the special effect image processing function, a control for triggering the special effect can be developed in advance, and when it is detected that the user triggers the control, the special effect triggering operation can be responded to start collecting the image to be processed. The image uploaded by the user can also be used as the image to be processed to process a certain image.

[0095] In the embodiments, the special effect triggering operation includes at least one of the following: triggering a style image generation prop; the voice information listened to includes a special effect adding instruction; and the display interface includes a face image.

[0096] The style image generation prop can be triggered by a key displayed on the display interface of the application software, and the triggering of the key represents that the current special effect image needs to be determined.

[0097] It can be understood that the voice information is collected based on the microphone array deployed on the terminal device, and the voice information is analyzed and processed. If the processing result includes a word for adding a special effect, it means that the special effect adding function is triggered. The advantage of determining whether to add a special effect based on the content of the voice information is that the interaction between the user and the display page is avoided, and the intelligence of special effect adding is improved. Another implementation manner can be that whether the face image of the user is included in the field of view range is determined according to the field of view range of the mobile terminal, and when the face image of the user is detected, the application software can take the detection of the face image as a triggering event of the special effect display operation. It should be understood by those skilled in the art that the specific event selected as the triggering event of the special effect display operation can be set according to the actual situation, and the embodiments of the present disclosure are not limited here.

[0098] Specifically, when the user triggers the special effect display operation, the image to be processed can be collected, so that the first style image generation model and the second style image generation model deployed in the mobile terminal process the image to be processed to obtain a target special effect image fused with the first style feature and the second style feature.

[0099] S520, determining a to-be-processed Gaussian noise corresponding to the to-be-processed image, processing the to-be-processed Gaussian noise based on the first feature fusion function corresponding to at least one convolution layer in the first style image generation model and the second feature fusion function of at least one convolution layer in the second style image generation model, and obtaining a target special effect image fused with the first style feature and the second style feature.

[0100] In this embodiment, the first style image generation model and the second style image generation model can be integrated in the corresponding application software based on a specific algorithm, and then the application software is deployed in the terminal device.

[0101] It should be noted that, since the first style image generation model and the second style image generation model are both trained after the model parameters of the face image generation model of the transfer learning, the model structures of the two models are the same, and each includes at least one convolution layer. The feature fusion function corresponding to each convolution layer can be preset. The feature fusion function is mainly used to determine how to fuse the features output by each convolution layer, and can realize the fusion of specific regions.

[0102] It should be noted that the model structures of the first style image generation model and the second style image generation model are completely the same, and each includes multiple convolution layers. For convolution layers of the same level, the first feature fusion function and the second feature fusion function are the same.

[0103] Specifically, the target compilation model can be deployed on the terminal device to convert the to-be-processed image into corresponding Gaussian noise. Further, each convolution layer in the trained first model can output features of different dimensions corresponding to the Gaussian noise, and each convolution layer in the trained second model can output features of different dimensions corresponding to the Gaussian noise. Based on the preset feature fusion function corresponding to each convolution layer, the features output by each convolution layer can be fused to obtain a result, which is used as the input of the next convolution layer until the last convolution layer outputs corresponding features and the features are fused based on the feature fusion function to obtain target features. The target features can be compiled to obtain a target special effect image.

[0104] In this embodiment, the feature dimensions output by different convolution layers are different, and the feature processing dimension corresponding to the convolution layer of the next level is greater than the feature processing dimension corresponding to the convolution layer of the previous level.

[0105] In order to more clearly understand the processing of the to-be-processed image by the second style image generation model, the first style image generation model and the feature fusion function corresponding to the convolution layer, a specific image can be used for illustration, as shown in Figure 6 .

[0106] Referring to Figure 6The first style image generation model is denoted as G2, and the second style image generation model is denoted as G3. The feature fusion function corresponding to each convolutional layer can be f i = (1 - a i )f 2 i + a i f 3 i wherein i represents the i-th convolutional layer, 2 represents the first style image generation model, 3 represents the second style image generation model, and the fusion parameter a i is a tensor with the same dimension as the current layer feature, which can realize feature region fusion by adjusting the parameter a i . In the current stage, a i of each layer can be a constant tensor, thereby realizing global feature fusion, but the value of a i of each layer can be adjusted, thereby realizing different fusion degrees at different scales. After obtaining the Gaussian noise Z of the image to be processed based on the target compilation sub-model, the Gaussian noise can be processed based on the full connection layer FC in the G2 model and the G3 model to obtain the processed feature. The processed feature is processed based on the first convolutional layer in the G2 model to obtain the output feature f 2 1, and the processed feature is processed based on the first convolutional layer in the G3 model to obtain the output feature f 3 1. The feature fusion function f1 based on the first convolutional layer is used to fuse f 2 1 and f 3 1 to obtain the fused feature. The fused feature is input into the second convolutional layer of the G2 model and the G3 model, respectively, to output f 2 2 and f 3 2, The feature fusion function f2 based on the second convolutional layer is used to fuse f 2 2 and f 3 2, to obtain the fused feature input into the third convolutional layer of the G2 model and the G3 model. The above steps are repeatedly executed until the last convolutional layer outputs f 2 i and f 3 i The feature fusion function f i based on the last convolutional layer is used to process f 2 i and f 3 i to obtain the target feature.

[0107] When the feature fusion is performed based on the above method, the accuracy of the feature fusion can be improved, that is, a iTo determine the specific features of the fusion, and then realize local effective fusion, so as to improve the effect of the target special effect image most suitable for the actual demand.

[0108] The above-mentioned feature fusion of the to-be-processed image based on the first style image generation model and the second style image generation model can be understood as: the to-be-processed Gaussian noise is processed based on at least one full connection layer in the first style image generation model and the second style image generation model to obtain the first feature and the second feature; the first feature is processed based on the convolution layer of the K level of the first style image generation model to obtain the first image feature; and the second feature is processed based on the convolution layer of the K level in the second style image generation model to obtain the second image feature; the first current feature and the second current feature are fused and processed based on the feature fusion function of the K level to obtain the fusion feature input into the convolution layer of the K+1 level; the fusion feature is input into the convolution layer of the K+1 level in the first style image generation model and the second style image generation model to update the first image feature and the second image feature, and the first image feature and the second image feature are fused and processed based on the feature fusion function of the K+1 level to update the fusion feature, until the fusion feature of the convolution layer of the last level of the first style image generation model and the second style image generation model is obtained.

[0109] The above-mentioned way of determining the target special effect image has the benefit of being able to clearly perform local feature fusion to obtain an image with corresponding style features, thereby improving the technical effect of the authenticity of the feature image.

[0110] The technical solution provided by the embodiments of the present disclosure can process the collected to-be-processed image to obtain a corresponding target special effect image by deploying the first style image generation model and the second style image generation model to the terminal device and setting the feature fusion functions corresponding to each convolution layer, and can well fuse the output features of the corresponding models, thereby improving the authenticity and realism of the special effect image.

[0111] Figure 7 For the flowchart of the image processing method provided by the embodiments of the present disclosure, in order to further improve the realism of the target special effect image, a training sample for fusing the first style feature and the second style feature can be constructed based on the first style image generation model and the second style image generation model, and then a third image generation model can be trained based on the training sample to replace the first image generation model and the second image generation model, and deployed on the terminal device to generate the target special effect image. The specific implementation can be referred to the detailed description of the technical solution. The same or corresponding technical terms as the above-mentioned embodiments are not repeated here.

[0112] AsFigure 7 The method comprises:

[0113] S710, determining a third training sample set based on at least one to-be-processed image and a corresponding target special effect image.

[0114] Obtaining at least one to-be-trained image; processing the at least one to-be-trained image based on a feature fusion function of at least one convolution layer in the first style image generation model and the second style image generation model to obtain a special effect image corresponding to each to-be-trained image; and determining a training sample based on each to-be-trained image and the corresponding special effect image.

[0115] It should be noted that the to-be-trained image can be an image collected based on an image collection device, or an image pre-stored in a storage space, or an image randomly generated by Gaussian noise.

[0116] It can be understood that after the to-be-processed image is collected, the to-be-trained image can be processed based on S510 and S520 to obtain a special effect image corresponding to each to-be-trained image, and a to-be-trained image and the corresponding special effect image at this time can be taken as a training sample. Since a large number of samples are required in the model training process, there are a large number of to-be-trained images, and correspondingly, there are also a large number of training samples.

[0117] S720, training a third to-be-trained image generation model based on the third training sample set to obtain a third image generation model.

[0118] It can be understood that the model parameters of the third to-be-trained image generation model are also determined by the model parameters of the face image generation model. Therefore, a small number of samples can be used to train the third to-be-trained image generation model to obtain the third image generation model. The samples at this time are target special effect images determined based on the first style image generation model, the second style image generation model, and the corresponding feature fusion function of the convolution layer. Therefore, the third training sample is a special effect image that fuses the first style feature and the second style feature, and the original face image. Correspondingly, the third image generation model can obtain a target special effect image that fuses the first style feature and the second style feature after processing the input to-be-processed image.

[0119] S730, replacing the first style image generation model and the second style image generation model with the third image generation model to process the to-be-processed image based on the third image generation model to obtain a target special effect image that fuses the first style feature and the second style feature.

[0120] Specifically, the third image generation model can be deployed on the mobile terminal device, or the third image generation model can be deployed in corresponding image processing software to install the image processing software on the terminal device. It can be understood that the first style image generation model and the second style image generation model can be replaced based on the third image generation model, so that when the to-be-processed image is collected, the target special effect image that fuses the first style feature and the second style feature can be generated based on the third image generation model.

[0121] The technical solution provided by the embodiments of the present disclosure can further reduce the memory occupation on the terminal device. The first style image generation model and the second style image generation model can be used to process the to-be-processed image to obtain a target special effect image corresponding to the to-be-processed image. The third training sample can be determined based on the target special effect image and the to-be-processed image. The third image generation model that fuses the first style feature and the second style feature can be trained based on the third training sample. The to-be-processed image can be processed based on the third image generation model deployed on the terminal device to obtain the target special effect image, which not only reduces the memory occupation, but also improves the authenticity and realism of the special effect image.

[0122] Figure 8 The flowchart of the image processing method provided by the embodiments of the present disclosure can further improve the matching of the target special effect image and the target object in the to-be-processed image. The third image generation model and the face image generation model can be deployed on the terminal device, and the face feature, the first style feature, and the second style feature of the to-be-processed image can be fused based on the feature fusion parameters corresponding to each convolution layer to obtain the target special effect image of the to-be-processed image. For specific implementation, please refer to the detailed description of the technical solution. The same or corresponding technical terms as the above embodiments are not repeated here.

[0123] As shown in Figure 8 The method comprises:

[0124] S810, deploying the target compiled model, the face image generation model, and the third image generation model on the terminal device.

[0125] Specifically, the output of the target compiled model is taken as the input of the face image generation model and the third image generation model according to the relationship between the input and the output, and the above models are deployed on the terminal device.

[0126] S820, processing the to-be-processed Gaussian noise based on the feature fusion parameters corresponding to the third image generation model, the face image generation model, and at least one convolution layer to obtain the target special effect image that fuses the first style feature, the second style feature, and the face image feature.

[0127] The feature fusion parameters corresponding to each convolutional layer can be pre-configured. Optionally, the feature fusion parameters can be a feature fusion function, and the feature fusion function has the same function and effect as the feature fusion function mentioned above, and thus will not be described herein.

[0128] For example, referring to Figure 9a and Figure 9b The face image generation model is denoted as G1, the target compilation model is denoted as E, and the third image generation model is denoted as G4. The collected image to be processed can be input into the target compilation model to obtain Gaussian noise of the image to be processed. The Gaussian noise is input into the face image generation models G1 and G4, respectively, and is processed based on the features output by each convolutional layer and the corresponding feature fusion function to obtain a target special effect image. The feature fusion function can extract and fuse the face image of the G1 model and the first feature and the second feature of the G4 model to obtain a target special effect image that is very similar to the face information of the image to be processed and that fuses the first style feature and the second style feature. The way in which the features output by each convolutional layer in the models G1 and G4 are processed based on the feature fusion function is the same as the processing method described above, and thus will not be described herein.

[0129] For example, referring to Figure 9b The image to be processed is input into the target compilation model to obtain processed Gaussian noise, and the processed Gaussian noise is input into the face image generation models G1 and G4, respectively, and is processed based on the features output by each convolutional layer and the corresponding feature fusion function to obtain a target special effect image that is similar to the face information of the image to be processed and that fuses the era feature and the ancient painting feature. For details, refer to Figure 9b .

[0130] The technical solution provided by the embodiments of the present disclosure can deploy the third image generation model and the face image generation model on a terminal device, can perform local feature fusion on the results output by the convolutional layers of each same layer based on the feature fusion function corresponding to the convolutional layers, and can obtain a target special effect image that fuses the face feature, the first style feature, and the second style feature. The face feature in the special effect image corresponds to the face feature extracted by the face image generation model, and at the same time, matches the first style feature and the second style feature. Therefore, the obtained target special effect image not only matches the face feature of the user in the image to be processed best, but also can well display the corresponding style features, thereby improving the technical effect of the realism of the target special effect image.

[0131] Figure 10 The structural block diagram of the image processing device provided by the embodiments of the present disclosure can execute the image processing method provided by any of the embodiments of the present disclosure, has the function modules and beneficial effects corresponding to the execution method. For example, Figure 10As shown, the device specifically comprises: an image acquisition module 1001 and a special effect image generation module 1002

[0132] The image acquisition module 1001 is configured to acquire a to-be-processed image; the special effect image generation module 1002 is configured to determine to-be-processed Gaussian noise corresponding to the to-be-processed image, process the to-be-processed Gaussian noise based on a first feature fusion function corresponding to at least one convolution layer in a first style image generation model and a second feature fusion function of at least one convolution layer in a second style image generation model, and obtain a target special effect image fused with a first style feature and a second style feature; the first style feature corresponds to the first style image generation model, and the second style feature corresponds to the second style image generation model.

[0133] On the basis of the above technical solution, the first style image generation model and the second style image generation model are both obtained by post-training of model parameters of a transfer learning face image generation model.

[0134] On the basis of the above technical solution, the special effect image generation module comprises a Gaussian noise determination unit configured to process the to-be-processed image based on a target compilation model to obtain Gaussian noise corresponding to the to-be-processed image.

[0135] On the basis of the above technical solution, the first style image generation model and the second style image generation model have the same model structure and both comprise a plurality of convolution layers, and the device further comprises:

[0136] A feature fusion function determination module is configured to determine a feature fusion function corresponding to convolution layers of the same level in the first style image generation model and the second style image generation model, so that when the convolution layers of the same level output features corresponding to the to-be-processed image respectively, the feature fusion function is used to fuse the features, to obtain fused features input to the next level, until the last level of convolution layer outputs the target special effect image.

[0137] On the basis of the above technical solution, the special effect image generation module comprises:

[0138] The feature extraction unit is configured to process the to-be-processed Gaussian noise based on at least one full connection layer in the first style image generation model and the second style image generation model to obtain first features and second features; the feature processing unit is configured to process the first features based on a convolution layer at a Kth level in the first style image generation model to obtain first image features, and process the second features based on a convolution layer at the Kth level in the second style image generation model to obtain second image features; the feature fusion unit is configured to fuse and process the first current features and the second current features based on a feature fusion function at the Kth level to obtain fused features input to a convolution layer at a K+1th level; and the feature updating unit is configured to input the fused features to the convolution layer at the K+1th level in the first style image generation model and the second style image generation model, respectively, to update the first image features and the second image features, and fuse and process the first image features and the second image features based on a feature fusion function at the K+1th level to update the fused features, until fused features of a convolution layer at a last level in the first style image generation model and the second style image generation model are obtained.

[0139] On the basis of the technical solution, the face image generation module is configured to train a to-be-trained image generator and a to-be-trained discriminator based on Gaussian noise to obtain the face image generation model.

[0140] On the basis of the technical solution, the face image generation module is further configured to process Gaussian noise based on the to-be-trained image generator to obtain to-be-discriminated images, discriminate the to-be-discriminated images and real face images based on the to-be-trained discriminator to obtain a reference loss value, correct model parameters in the to-be-trained image generator based on the reference loss value, and obtain the face image generation model by converging loss functions in the to-be-trained image generator and the to-be-trained discriminator.

[0141] On the basis of the technical solution, the device further includes a training model construction module configured to obtain model parameters in the face image generation model and construct a first to-be-trained model and a second to-be-trained model based on the model parameters; a first style model determination module configured to train the first to-be-trained model based on training samples in a first style feature sample set to obtain a first style image generation model; and a second style model determination module configured to train the second to-be-trained model based on training samples in a second style feature sample set to obtain a second style image generation model; wherein the training samples include face images and theoretical output images of corresponding style features.

[0142] On the basis of the technical solution above, the device further comprises a compiling model determination module configured to train a to-be-trained compiling model based on the face image generation model and at least one face image, and obtain a target compiling model; wherein the target compiling model is configured to process an input face image into a corresponding Gaussian noise.

[0143] On the basis of the technical solution above, the first style feature corresponds to an era feature, and the second style feature is a traditional painting feature.

[0144] The technical solution provided by the embodiments of the present disclosure can process the collected to-be-processed image to obtain a corresponding target special effect image by deploying the first style image generation model and the second style image generation model on a terminal device and setting the feature fusion functions corresponding to the respective convolution layers, can well fuse the output features of the corresponding models, and further improve the authenticity and realism of the special effect image.

[0145] The device for image processing provided by the embodiments of the present disclosure can execute the method for image processing provided by any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of executing the method.

[0146] It is worth noting that each unit and module included in the device above is only divided according to the function logic, but is not limited to the division above, as long as the corresponding functions can be implemented; in addition, the specific names of each functional unit are only for easy mutual differentiation, and do not serve to limit the protection scope of the embodiments of the present disclosure.

[0147] Figure 11 The structural block diagram of the device for image processing provided by the embodiments of the present disclosure can execute the method for image processing provided by any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of executing the method. As shown in Figure 11 The device specifically comprises a Gaussian noise processing module 2001 and a special effect image determination module 2002.

[0148] The Gaussian noise processing module 2001 is configured to collect a to-be-processed image and determine a to-be-processed Gaussian noise corresponding to the to-be-processed image; and the special effect image determination module 2002 is configured to process the to-be-processed Gaussian noise based on a third image generation model to obtain a target special effect image fused with a first style feature and a second style feature; wherein the third image generation model is obtained by training a training sample based on a first feature fusion function of at least one convolution layer in a first style image generation model and a second feature fusion function of at least one convolution layer in a second style image generation model, the first style image generation model corresponds to the first style feature, and the second style image generation model corresponds to the second style feature.

[0149] On the basis of the technical solution, the special effect image generation module comprises:

[0150] The image acquisition unit is configured to acquire at least one training image, wherein the training image comprises a face image; the image processing unit is configured to process the at least one training image based on a feature fusion function of at least one convolution layer in the first style image generation model and the second style image generation model, to obtain a special effect image corresponding to each training image; and the sample construction unit is configured to determine the training sample based on each training image and the corresponding special effect image.

[0151] On the basis of the technical solution, the device further comprises:

[0152] The model training module is configured to train a third training image generation model based on the training sample, to obtain the third face image generation model; wherein the model parameters of the third training image generation model are the same as those of the face image generation model.

[0153] On the basis of the technical solution, the special effect image generation module is further configured to process the to-be-processed Gaussian noise based on a first feature fusion function of at least one convolution layer in the third image generation model and a second feature fusion function of at least one convolution layer in the face image generation model, to obtain a target special effect image in which the first style feature, the second style feature and the face image feature are fused.

[0154] On the basis of the technical solution, the first style feature corresponds to an era feature, and the second style feature is an ancient painting feature.

[0155] The technical solution provided by the embodiments of the present disclosure can process the collected to-be-processed image to obtain a corresponding target special effect image by deploying the first style image generation model and the second style image generation model to the terminal device and setting the feature fusion functions corresponding to each convolution layer, can well fuse the output features of the corresponding model, and can further improve the authenticity and realism of the special effect image.

[0156] The device for image processing provided by the embodiments of the present disclosure can execute the method for image processing provided by any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of the execution method.

[0157] It should be noted that each unit and module included in the device is only divided according to the function logic, but is not limited to the above division, as long as the corresponding function can be realized; in addition, the specific name of each functional unit is only for convenient mutual distinction, and does not serve to limit the protection scope of the embodiments of the present disclosure.

[0158] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Reference is made below. Figure 12 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 12 The diagram below shows the structure of the terminal device or server 1100. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 12 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0159] like Figure 12 As shown, electronic device 1100 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1101, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1102 or a program loaded from storage device 1108 into random access memory (RAM) 1103. The RAM 1103 also stores various programs and data required for the operation of electronic device 1100. The processing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. An edit / output (I / O) interface 1105 is also connected to bus 1104.

[0160] Typically, the following devices can be connected to I / O interface 1105: input devices 1106 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1107 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1108 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1109. Communication device 1109 allows electronic device 1100 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 12 An electronic device 1100 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0161] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication apparatus 1109, or installed from the storage apparatus 1108, or installed from the ROM 1102. When the computer program is executed by the processing apparatus 1101, the above-mentioned functions defined in the methods of embodiments of the present disclosure are executed.

[0162] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0163] The electronic device provided by the embodiments of the present disclosure and the image processing method provided by the above-mentioned embodiments belong to the same inventive concept, and the technical details not described in detail in the present embodiment can be referred to the above-mentioned embodiments, and the present embodiment has the same beneficial effects as the above-mentioned embodiments.

[0164] The embodiments of the present disclosure provide a computer storage medium, which stores a computer program, and the program is executed by a processor to implement the image processing method provided by the above-mentioned embodiments.

[0165] It should be noted that the computer-readable medium in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, a RF (radio frequency), or the like, or any suitable combination thereof.

[0166] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0167] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and not be assembled into the electronic device.

[0168] The computer-readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to:

[0169] Collect an image to be processed;

[0170] determining to-be-processed Gaussian noise corresponding to the to-be-processed image, processing the to-be-processed Gaussian noise based on a first feature fusion function corresponding to at least one convolutional layer in a first style image generation model and a second feature fusion function of at least one convolutional layer in a second style image generation model, and obtaining a target special effect image fused with first style features and second style features.

[0171] The first style features correspond to the first style image generation model, and the second style features correspond to the second style image generation model.

[0172] Or,

[0173] acquiring a to-be-processed image and determining to-be-processed Gaussian noise corresponding to the to-be-processed image;

[0174] processing the to-be-processed Gaussian noise based on a third image generation model to obtain a target special effect image fused with first style features and second style features;

[0175] The third image generation model is trained based on a first feature fusion function of at least one convolutional layer in a first style image generation model and a second feature fusion function of at least one convolutional layer in a second style image generation model, the first style image generation model corresponds to first style features, and the second style image generation model corresponds to second style features.

[0176] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0177] The computer program product of the first aspect can include one or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform the operations of the first aspect. The computer program product of the first aspect can include a non-transitory computer-readable medium storing code that, when executed, causes a computer to perform operations for the first aspect.

[0178] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the names of the units do not constitute a limitation on the units themselves. For example, the first obtaining unit can also be described as a unit that obtains at least two Internet protocol addresses.

[0179] The functions described above in the detailed description of the above embodiments can be performed by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0180] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0181] The above description merely illustrates the preferred embodiments of the disclosure and a principle for applying the technologies. It is understood by those skilled in the art that the disclosed scope of the disclosure is not limited to the technical solutions formed by the specific combinations of the technical features described above, and should also cover other technical solutions formed by the combinations of the technical features described above or their equivalent features without departing from the disclosed concept. For example, the technical solutions formed by the mutual replacement of the above-described features and the technical features with similar functions disclosed in the disclosure (but not limited to) can be used.

[0182] Further, although operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order illustrated or sequentially. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, although specific implementation details are included for the purpose of providing a thorough disclosure, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0183] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. An image processing method, characterized in that, include: Acquire the image to be processed; The Gaussian noise to be processed is determined to correspond to the image to be processed. Based on the first feature fusion function corresponding to at least one convolutional layer in the first style image generation model and the second feature fusion function of at least one convolutional layer in the second style image generation model, the Gaussian noise to be processed is processed to obtain a target special effect image that fuses the first style features and the second style features. The first feature fusion function is the same as the second feature fusion function. The feature fusion function is used to perform feature fusion processing on the image features output by the convolutional layers of the same level in the first style image generation model and the second style image generation model. Wherein, the first style feature corresponds to the first style image generation model, and the second style feature corresponds to the second style image generation model.

2. The method according to claim 1, characterized in that, Both the first style image generation model and the second style image generation model are trained by transferring the model parameters of the face image generation model.

3. The method according to claim 1, characterized in that, The step of determining the Gaussian noise to be processed corresponding to the image to be processed includes: The image to be processed is processed based on the target compilation model to obtain Gaussian noise corresponding to the image to be processed.

4. The method according to claim 1, characterized in that, The first style image generation model and the second style image generation model have the same model structure and both include multiple convolutional layers. The method further includes: The feature fusion function corresponding to the convolutional layer at the same level in the first style image generation model and the second style image generation model is determined. When the convolutional layer at the same level outputs features corresponding to the image to be processed, the feature fusion is performed based on the feature fusion function to obtain the fusion features input to the next level, until the last level of the convolutional layer outputs the target special effect image. In this case, the first feature fusion function and the second feature fusion function corresponding to the same level of convolutional layer are the same.

5. The method according to claim 4, characterized in that, The processing of the Gaussian noise based on the first feature fusion function corresponding to at least one convolutional layer in the first style image generation model and the second feature fusion function of at least one convolutional layer in the second style image generation model includes: The Gaussian noise to be processed is processed based on at least one fully connected layer in the first style image generation model and the second style image generation model to obtain the first feature and the second feature; The first feature is processed by the Kth level convolutional layer of the first style image generation model to obtain the first image feature; and the second feature is processed by the Kth level convolutional layer of the second style image generation model to obtain the second image feature. The first image features and the second image features are fused based on the feature fusion function of the Kth level to obtain the fused features of the input convolutional layer at the (K+1)th level. The fused features are respectively input into the convolutional layer of the (K+1)th level in the first style image generation model and the second style image generation model to update the first image features and the second image features. The first image features and the second image features are fused based on the feature fusion function of the (K+1)th level convolutional layer to update the fused features until the fused features of the last level convolutional layer of the first style image generation model and the second style image generation model are obtained.

6. The method according to claim 2, characterized in that, Also includes: A facial image generation model is obtained by training the image generator and the discriminator under training using Gaussian noise.

7. The method according to claim 6, characterized in that, The process of training the image generator and the discriminator based on Gaussian noise to obtain the facial image generation model includes: The Gaussian noise is processed by the image generator to be trained to obtain the image to be judged; Based on the discriminator to be trained, the image to be discriminated and the real face image are discriminated to obtain a baseline loss value, and the model parameters in the image generator to be trained are corrected based on the baseline loss value. The convergence of the loss functions in the image generator and the discriminator to be trained is used as the training objective to obtain the facial image generation model.

8. The method according to claim 6, characterized in that, Also includes: Obtain the model parameters in the facial image generation model, and construct a first training model and a second training model based on the model parameters; The first model to be trained is trained based on the training samples in the first style feature sample set to obtain the first style image generation model. as well as, The second model to be trained is trained based on the training samples in the second style feature sample set to obtain the second style image generation model. The training samples include facial images and theoretical output images of corresponding style features.

9. The method according to claim 2, characterized in that, Also includes: Based on the facial image generation model and at least one facial image, the training compilation model is trained to obtain the target compilation model; wherein, the target compilation model is used to process the input facial image into corresponding Gaussian noise.

10. The method according to any one of claims 1-9, characterized in that, The first stylistic feature corresponds to the characteristics of the era, while the second stylistic feature is a characteristic of ancient paintings.

11. An image processing method, characterized in that, include: Acquire the image to be processed and determine the Gaussian noise to be processed corresponding to the image to be processed; The Gaussian noise to be processed is processed based on the third image generation model to obtain a target special effect image that integrates the first style features and the second style features; The third image generation model is trained on training samples constructed based on the first feature fusion function of at least one convolutional layer in the first style image generation model and the second feature fusion function of at least one convolutional layer in the second style image generation model. The first style image generation model corresponds to the first style feature, and the second style image generation model corresponds to the second style feature. The first feature fusion function is the same as the second feature fusion function. The feature fusion function is used to perform feature fusion processing on the image features output by the convolutional layers of the same level in the first style image generation model and the second style image generation model.

12. The method according to claim 11, characterized in that, Constructing training samples corresponding to the third image generation model includes: Acquire at least one training image, wherein the training image includes a facial image; Based on the feature fusion function of at least one convolutional layer in the first style image generation model and the second style image generation model, the at least one training image is processed to obtain special effect images corresponding to each training image; The training samples are determined based on each image to be trained and the corresponding special effects image.

13. The method according to claim 12, characterized in that, Also includes: The third image generation model is trained based on the training samples to obtain the third image generation model; The third image generation model to be trained has the same model parameters as the facial image generation model.

14. The method according to claim 12, characterized in that, The process of processing the Gaussian noise based on the third image generation model to obtain a target effect image that fuses the first style features and the second style features includes: Based on the first feature fusion function of at least one convolutional layer in the third image generation model and the second feature fusion function of at least one convolutional layer in the facial image generation model, the Gaussian noise to be processed is processed to obtain a target special effect image that integrates the first style feature, the second style feature and the facial image feature.

15. The method according to any one of claims 11-14, characterized in that, The first stylistic feature corresponds to the characteristics of the era, while the second stylistic feature is a characteristic of ancient paintings.

16. An image processing apparatus, characterized in that, include: The image acquisition module is used to acquire images to be processed. A special effects image generation model is used to determine the Gaussian noise to be processed corresponding to the image to be processed. Based on the first feature fusion function corresponding to at least one convolutional layer in the first style image generation model and the second feature fusion function of at least one convolutional layer in the second style image generation model, the Gaussian noise to be processed is processed to obtain a target special effects image that fuses the first style features and the second style features. The first feature fusion function is the same as the second feature fusion function. The feature fusion function is used to perform feature fusion processing on the image features output by the convolutional layers of the same level in the first style image generation model and the second style image generation model.

17. An image processing apparatus, characterized in that, include: A Gaussian noise processing module is used to acquire an image to be processed and determine the Gaussian noise to be processed corresponding to the image to be processed. The special effects image determination module is used to process the Gaussian noise to be processed based on the third image generation model to obtain a target special effects image that integrates the first style features and the second style features; The third image generation model is trained on training samples constructed based on the first feature fusion function of at least one convolutional layer in the first style image generation model and the second feature fusion function of at least one convolutional layer in the second style image generation model. The first style image generation model corresponds to the first style feature, and the second style image generation model corresponds to the second style feature. The first feature fusion function is the same as the second feature fusion function. The feature fusion function is used to perform feature fusion processing on the image features output by the convolutional layers of the same level in the first style image generation model and the second style image generation model.

18. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1-10 or 11-15.

19. A storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to perform the image processing method as described in any one of claims 1-10 or 11-15.

Images