Image style transfer model training method, image style transfer method and device

By learning and filtering target style images with consistent similarity step by step, the realism of the high-resolution image style transfer model is improved, and the problem of insufficient realism of high-resolution style images in the existing technology is solved.

CN113989104BActive Publication Date: 2024-11-05BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202111243493.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-25
Publication Date
2024-11-05
Estimated Expiration
2041-10-25

AI Technical Summary

Technical Problem

The existing high-resolution image style transfer methods have weak realism when processing image details, mainly because the low-resolution style conversion relationship is difficult to adapt to high resolution, resulting in the lack of details and insufficient realism of high-resolution style images.

Method used

By acquiring sample images, including low-resolution and high-resolution images, gradually increase image resolution, use neural network models for step-by-step learning, obtain the first image style transfer model and the target image style transfer model, filter similarity and style Consistent target style images, trained multiple times to improve image detail fitting ability.

Benefits of technology

It improves the realism of high-resolution style images, avoids the problem of insufficient realism caused by high-resolution style conversion relationships, and enhances the accuracy and authenticity of image details processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113989104B_ABST
    Figure CN113989104B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a training method, an image style transfer method, an image style transfer method, an apparatus, an electronic device and a storage medium for an image style transfer model. The training method for the image style transfer model comprises: obtaining a sample image; the sample image comprises at least a first sample image and a second sample image, the first sample image being obtained by downsampling the second sample image; obtaining a first target style image corresponding to the first sample image, training a first neural network model according to the first sample image and the first target style image to obtain a first image style transfer model; obtaining a second target style image corresponding to the second sample image according to the first image style transfer model; training a second neural network model according to the second sample image and the second target style image to obtain a target image style transfer model. The adoption of this method is conducive to improving the realism of the obtained high-resolution style image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a training method for an image style transfer model, an image style transfer method, a device, an electronic device, and a storage medium. Background Art

[0002] Image style transfer refers to changing the details and style of an image while retaining the main content of the image, such as changing the hair color and skin color of a portrait. This technology can be widely used in image processing, computer image synthesis, and computer vision.

[0003] In the related art, the current high-resolution image style transfer method, when a high-resolution style map is difficult to obtain, generally uses a low-resolution original image and its corresponding style map to calculate the function conversion relationship of the image style, and applies the function conversion relationship to the high-resolution original image to obtain a high-resolution style map; however, the low-resolution style transfer relationship is different from the high-resolution style transfer relationship, and there are many defects when processing image details, resulting in a high-resolution style map with more missing image details and weaker sense of reality. Summary of the invention

[0004] The present disclosure provides a training method for an image style transfer model, an image style transfer method, an apparatus, an electronic device, and a storage medium, to at least solve the problem that the high-resolution style map obtained in the related art has a weak sense of reality. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for training an image style transfer model is provided, comprising:

[0006] Acquire a sample image; the sample image includes at least a first sample image and a second sample image, and the first sample image is obtained by downsampling the second sample image;

[0007] Acquire a first target style image corresponding to the first sample image, and train a first neural network model according to the first sample image and the first target style image to obtain a first image style transfer model;

[0008] Obtaining a second target style image corresponding to the second sample image according to the first image style transfer model;

[0009] A second neural network model is trained according to the second sample image and the second target style image to obtain a target image style transfer model.

[0010] In an exemplary embodiment, obtaining a second target style image corresponding to the second sample image according to the first image style transfer model includes:

[0011] Inputting the second sample image into the first image style transfer model to perform image style transfer processing to obtain a second target style image corresponding to the second sample image;

[0012] The step of training a second neural network model according to the second sample image and the second target style image to obtain a target image style transfer model includes:

[0013] Inputting the second sample image into a second neural network model to perform image style transfer processing to obtain a second predicted style image corresponding to the second sample image;

[0014] Obtaining a loss value according to a difference between the second target style image and the second predicted style image;

[0015] The second neural network model is trained according to the loss value to obtain the target image style transfer model.

[0016] In an exemplary embodiment, the sample images further include N third sample images, and the N third sample images are obtained by downsampling the second sample images, or the N third sample images are obtained by upsampling the first sample images; the image resolutions of the first sample image, the N third sample images, and the second sample image increase in sequence; N is a positive integer;

[0017] The step of obtaining a second target style image corresponding to the second sample image according to the first image style transfer model further includes:

[0018] In order of image resolution from low to high, taking the first third sample image among the N third sample images as the current third sample image, inputting the first image style transfer model to perform image style transfer processing, and obtaining a third target style image corresponding to the current third sample image;

[0019] According to the current third sample image and the third target style image corresponding to the current third sample image, training a corresponding current third neural network model among the N third neural network models to obtain a current third image style transfer model;

[0020] Taking the next third sample image of the current third sample image among the N third sample images as the current third sample image, inputting the current third image style transfer model into the current third image to perform image style transfer processing, and obtaining a third target style image corresponding to the current third sample image;

[0021] Retraining the corresponding current third neural network model among the N third neural network models according to the current third sample image and the third target style image corresponding to the current third sample image, until the obtained current third image style transfer model is the Nth third image style transfer model;

[0022] The second sample image is input into the Nth third image style transfer model to perform image style transfer processing to obtain a second target style image corresponding to the second sample image.

[0023] In an exemplary embodiment, the current third image style transfer model is trained in the following manner:

[0024] From the third target style images corresponding to the current third sample image, a third target style image having an image content similarity with the current third sample image greater than a first preset similarity and having the same target conversion style as a previous third image style transfer model of the current third image style transfer model is selected as the third target style image to be referenced;

[0025] Filtering out, from the current third sample images, a third sample image corresponding to the third target style image to be referenced as a third target sample image;

[0026] According to the third target sample image and the third target style image corresponding to the third target sample image, the corresponding current third neural network model among the N third neural network models is trained to obtain the current third image style transfer model.

[0027] In an exemplary embodiment, after inputting the second sample image into the Nth third image style transfer model for image style transfer processing to obtain a second target style image corresponding to the second sample image, the method further includes:

[0028] From the second target style image, a second target style image having an image content similarity with the second sample image greater than a second preset similarity and having the same target conversion style as the Nth third image style transfer model is selected as the second target style image to be referenced;

[0029] The step of training a second neural network model according to the second sample image and the second target style image to obtain a target image style transfer model further includes:

[0030] Filtering out, from the second sample images, a second sample image corresponding to the second target style image to be referenced as a second target sample image;

[0031] The second neural network model is trained according to the second target sample image and the second target style image corresponding to the second target sample image to obtain the target image style transfer model.

[0032] According to a second aspect of an embodiment of the present disclosure, there is provided an image style transfer method, comprising:

[0033] Get the image to be processed;

[0034] The image to be processed is input into a target image style transfer model to obtain a target style image corresponding to the image to be processed; the target image style transfer model is obtained according to the training method of the image style transfer model described in any one of the embodiments of the first aspect.

[0035] According to a third aspect of an embodiment of the present disclosure, a training device for an image style transfer model is provided, comprising:

[0036] A first acquisition unit is configured to acquire a sample image; the sample image includes at least a first sample image and a second sample image, and the first sample image is obtained by downsampling the second sample image;

[0037] A first training unit is configured to obtain a first target style image corresponding to the first sample image, and train a first neural network model according to the first sample image and the first target style image to obtain a first image style transfer model;

[0038] A second acquisition unit is configured to execute the first image style transfer model to obtain a second target style image corresponding to the second sample image;

[0039] The second training unit is configured to train a second neural network model according to the second sample image and the second target style image to obtain a target image style transfer model.

[0040] In an exemplary embodiment, the second acquisition unit is further configured to input the second sample image into the first image style transfer model to perform image style transfer processing to obtain a second target style image corresponding to the second sample image;

[0041] The second training unit is further configured to input the second sample image into a second neural network model for image style transfer processing to obtain a second predicted style image corresponding to the second sample image; obtain a loss value based on the difference between the second target style image and the second predicted style image; and train the second neural network model based on the loss value to obtain the target image style transfer model.

[0042] In an exemplary embodiment, the sample images further include N third sample images, and the N third sample images are obtained by downsampling the second sample images, or the N third sample images are obtained by upsampling the first sample images; the image resolutions of the first sample image, the N third sample images, and the second sample image increase in sequence; N is a positive integer;

[0043] The second acquisition unit is further configured to execute, in order of image resolution from low to high, taking the first third sample image of the N third sample images as the current third sample image, inputting the first image style transfer model to perform image style transfer processing, and obtaining a third target style image corresponding to the current third sample image; training the corresponding current third neural network model of the N third neural network models according to the current third sample image and the third target style image corresponding to the current third sample image, and obtaining a current third image style transfer model; taking the next third sample image of the current third sample image of the N third sample images as the current third sample image again, inputting the current third image style transfer model to perform image style transfer processing, and obtaining a third target style image corresponding to the current third sample image; retraining the corresponding current third neural network model of the N third neural network models according to the current third sample image and the third target style image corresponding to the current third sample image, until the obtained current third image style transfer model is the Nth third image style transfer model; inputting the second sample image into the Nth third image style transfer model to perform image style transfer processing, and obtaining a second target style image corresponding to the second sample image.

[0044] In an exemplary embodiment, the second acquisition unit is further configured to perform, from the third target style image corresponding to the current third sample image, screening out a third target style image whose similarity with the image content of the current third sample image is greater than a first preset similarity and has the same target conversion style as the previous third image style transfer model of the current third image style transfer model, as the third target style image to be referenced; screening out a third sample image corresponding to the third target style image to be referenced from the current third sample image, as the third target sample image; and training the corresponding current third neural network model among the N third neural network models according to the third target sample image and the third target style image corresponding to the third target sample image, to obtain the current third image style transfer model.

[0045] In an exemplary embodiment, the apparatus further includes an image screening unit configured to screen out, from the second target style image, a second target style image having an image content similarity with the second sample image greater than a second preset similarity and having the same target conversion style as the Nth third image style transfer model, as the second target style image to be referenced;

[0046] The second training unit is further configured to screen out a second sample image corresponding to the second target style image to be referenced from the second sample image as the second target sample image; and train the second neural network model according to the second target sample image and the second target style image corresponding to the second target sample image to obtain the target image style transfer model.

[0047] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image style transfer apparatus, comprising:

[0048] An acquisition unit, configured to acquire an image to be processed;

[0049] A processing unit is configured to input the image to be processed into a target image style transfer model to obtain a target style image corresponding to the image to be processed; the target image style transfer model is obtained according to the training method of the image style transfer model described in any one of the embodiments of the first aspect.

[0050] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the training method of the image style transfer model as described in any one of the embodiments of the first aspect, and the image style transfer method as described in any one of the embodiments of the second aspect.

[0051] According to a sixth aspect of an embodiment of the present disclosure, a storage medium is provided, comprising: when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the image style transfer model described in any one of the embodiments of the first aspect, and the image style transfer method described in any one of the embodiments of the second aspect.

[0052] According to a seventh aspect of the embodiments of the present disclosure, a computer program product is provided, wherein the computer program product includes instructions, and when the instructions are executed by a processor of an electronic device, the electronic device is capable of executing the training method of the image style transfer model described in any one of the embodiments of the first aspect, and the image style transfer method described in any one of the embodiments of the second aspect.

[0053] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0054] By obtaining a sample image; the sample image includes at least a first sample image and a second sample image, the first sample image is obtained by downsampling the second sample image; then obtaining a first target style image corresponding to the first sample image, training a first neural network model according to the first sample image and the first target style image, and obtaining a first image style transfer model; then obtaining a second target style image corresponding to the second sample image according to the first image style transfer model; finally, training a second neural network model according to the second sample image and the second target style image, and obtaining a target image style transfer model. In this way, in the process of gradually increasing the image resolution of the sample image, the image style transfer model can fit the image details well through step-by-step learning, thereby obtaining a more realistic detail processing result, so that the style image output by the target image style transfer model obtained by the final training has a stronger sense of reality, thereby improving the sense of reality of the obtained high-resolution style image, and avoiding the defect that the low-resolution style transfer relationship is difficult to adapt to the high resolution, resulting in a weaker sense of reality of the obtained high-resolution style image.

[0055] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0057] Figure 1 The figure is an application environment diagram of a training method for an image style transfer model according to an exemplary embodiment.

[0058] Figure 2 The present invention is a flowchart of a method for training an image style transfer model according to an exemplary embodiment.

[0059] Figure 3 is a flowchart showing steps of training a second neural network model according to an exemplary embodiment.

[0060] Figure 4 The figure is a flowchart of a method for image style transfer according to an exemplary embodiment.

[0061] Figure 5 The present invention is a flowchart of a method for high-resolution texture style conversion based on model self-learning according to an exemplary embodiment.

[0062] Figure 6 The figure is a block diagram of a training device for an image style transfer model according to an exemplary embodiment.

[0063] Figure 7 The figure is a block diagram of an image style transfer apparatus according to an exemplary embodiment.

[0064] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0065] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0066] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0067] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0068] The training method of the image style transfer model provided in the present disclosure can be applied to Figure 1 In the application environment shown. Figure 1 The application environment diagram includes a terminal 110, which is an electronic device with an image style transfer function. The electronic device may be a smart phone, a tablet computer, a laptop computer, a personal computer, or the like. Figure 1 The terminal 110 is a smart phone as an example for explanation. Figure 1, the terminal 110 obtains a sample image; the sample image includes at least a first sample image and a second sample image, and the first sample image is obtained by downsampling the second sample image; a first target style image corresponding to the first sample image is obtained, and a first neural network model is trained according to the first sample image and the first target style image to obtain a first image style transfer model; a second target style image corresponding to the second sample image is obtained according to the first image style transfer model; a second neural network model is trained according to the second sample image and the second target style image to obtain a target image style transfer model.

[0069] Figure 2 is a flowchart of a method for training an image style transfer model according to an exemplary embodiment. Figure 2 As shown, the training method of the image style transfer model is used as follows Figure 1 In the terminal shown, the following steps are included:

[0070] In step S210, a sample image is obtained; the sample image includes at least a first sample image and a second sample image, and the first sample image is obtained by downsampling the second sample image.

[0071] Among them, the sample image refers to the original image used for model training without image style transfer, such as the original portrait image (including hair), which can be taken by a mobile phone, camera, camcorder or other shooting equipment, or obtained from the Internet or local image library through a terminal.

[0072] The image resolution of the first sample image is smaller than that of the second sample image, and the image content of the first sample image is the same as that of the second sample image. Specifically, the first sample image refers to an image with low image resolution, such as a 1K resolution portrait image (including hair), which can be obtained by downsampling the second sample image; the second sample image refers to an image with high image resolution, such as a 4K resolution portrait image (including hair), which can be directly acquired by the terminal.

[0073] It should be noted that the present disclosure involves multiple first sample images and multiple second sample images.

[0074] Specifically, the terminal may obtain an image with high image resolution that has not undergone image style transfer from the network or a local image library as the second sample image; and obtain an image after downsampling as the first sample image by downsampling the second sample image.

[0075] For example, the terminal collects a high-resolution portrait image without hair dyeing in advance as the second sample image; and obtains a low-resolution portrait image without hair dyeing as the first sample image by downsampling the second sample image.

[0076] In step S220, a first target style image corresponding to the first sample image is obtained, and a first neural network model is trained according to the first sample image and the first target style image to obtain a first image style transfer model.

[0077] Among them, the target style image refers to an image with a different style effect from the sample image, but the main content of the image is the same as the sample image, such as a face image with a hair color change, a face image with a skin color change, etc.; the first target style image refers to a low-resolution target style image corresponding to the first sample image, such as a low-resolution yellow-haired face image, a low-resolution black-skinned face image, etc., which can be obtained by manual image editing by designers or processing by a special image style transfer model. It should be noted that the style involved in this disclosure does not refer to the style of painting schools such as abstractionism and expressionism, but refers to the texture, color, visual pattern, etc. of a picture in a narrow sense.

[0078] Among them, the first neural network model refers to a deep neural network model; the first image style transfer model refers to a low-resolution image style transfer model used for image style transfer, specifically refers to a low-resolution image style transfer model used to perform color conversion on the hair area in a face image.

[0079] Specifically, the terminal performs image style transfer processing on the first sample image to obtain a target style image corresponding to the first sample image, and uses it as the first target style image; inputs the first sample image into the first neural network model, performs image style transfer processing on the first sample image through the first neural network model, and obtains a first predicted style image corresponding to the first sample image; obtains a first loss value of the first neural network model based on the difference between the first target style image and the first predicted style image; and adjusts the model parameters of the first neural network model based on the first loss value to obtain a trained first neural network model as the first image style transfer model.

[0080] For example, the terminal performs difference processing on the first target style image and the first predicted style image to obtain the difference between the first target style image and the first predicted style image, and combines the loss function to obtain the first loss value of the first neural network model; if the first loss value is greater than or equal to the first preset threshold, the model parameters of the first neural network model are adjusted according to the first loss value to obtain the first neural network model after the model parameters are adjusted; and the above process is repeated to repeatedly train the first neural network model after the model parameters are adjusted, until the first loss value obtained according to the trained first neural network model is less than the first preset threshold, and the trained first neural network model is used as the first image style transfer model.

[0081] In actual scenarios, designers adjust the color of the dyed hair area in the low-resolution face image to obtain a low-resolution face image of the real hair dyed hair, or use a special image style transfer model to convert the color of the dyed hair area in the low-resolution face image to obtain a low-resolution face image of the real hair dyed hair; then, the terminal inputs the low-resolution face image into the first neural network model to obtain a predicted low-resolution face image after hair dyeing; based on the low-resolution face image of the real hair dyed hair and the predicted low-resolution face image after hair dyeing, the model parameters in the first neural network model are optimized based on the random gradient descent method to reduce the error between the predicted low-resolution face image after hair dyeing and the real low-resolution face image after hair dyeing; the above process is repeated, and after a preset number of iterations, the iteration is terminated, and the first neural network model finally obtained is used as the first image style transfer model.

[0082] In step S230, a second target style image corresponding to the second sample image is obtained according to the first image style transfer model.

[0083] Among them, the second target style image refers to a high-resolution target style image corresponding to the second sample image, such as a high-resolution yellow-haired portrait image, a high-resolution black-skin portrait image, etc., which can be obtained by performing image style transfer processing on the second sample image through the first image style transfer model, or by performing image style transfer processing on the second sample image through a medium-resolution image style transfer model obtained based on the first image style transfer model.

[0084] Specifically, the terminal inputs the second sample image into the first image style transfer model, performs image style transfer processing on the second sample image through the first image style transfer model, obtains a target style image corresponding to the second sample image, and uses it as a second target style image corresponding to the second sample image.

[0085] For example, in an actual scenario, the terminal uses a low-resolution image style transfer model for color conversion of the hair area in the face image to convert the color of the dyed hair area in the high-resolution face image, and obtains a high-resolution face image with real dyed hair. In other words, in this way, a large number of high-resolution face images with real dyed hair can be quickly created.

[0086] Furthermore, the terminal may also obtain a third sample image; the third sample image is obtained by downsampling the second sample image, or the third sample image is obtained by upsampling the first sample image, and refers to an image with medium image resolution, such as a 2K resolution portrait image (including hair); the image resolutions of the first sample image, the third sample image, and the second sample image increase in sequence; then, the terminal inputs the third sample image into the first image style transfer model, performs image style transfer processing on the third sample image through the first image style transfer model, obtains a target style image corresponding to the third sample image, and uses it as a third target style image corresponding to the third sample image; wherein the third target style image refers to a medium resolution target style image corresponding to the third sample image, Such as medium-resolution yellow-haired portrait pictures, medium-resolution black-skin portrait pictures, etc.; then, the terminal trains a third neural network model (such as a deep neural network model) according to the third sample image and the third target style image to obtain a third image style transfer model; wherein the third image style transfer model refers to a medium-resolution image style transfer model for image style transfer, specifically refers to a medium-resolution image style transfer model for color conversion of hair areas in face images; finally, the terminal inputs the second sample image into the third image style transfer model, performs image style transfer processing on the second sample image through the third image style transfer model, obtains a target style image corresponding to the second sample image, and uses it as the second target style image corresponding to the second sample image.

[0087] In step S240, the second neural network model is trained according to the second sample image and the second target style image to obtain a target image style transfer model.

[0088] Among them, the second neural network model also refers to a deep neural network model; the target image style transfer model refers to a high-resolution image style transfer model used for image style transfer, specifically refers to a high-resolution image style transfer model used to perform color conversion on the hair area in the face image.

[0089] Specifically, the terminal inputs the second sample image into the second neural network model, performs image style transfer processing on the second sample image through the second neural network model, and obtains a predicted style image corresponding to the second sample image; obtains a second loss value of the second neural network model based on the difference between the second target style image and the second predicted style image; and adjusts the model parameters of the second neural network model based on the second loss value to obtain a trained second neural network model as a target image style transfer model.

[0090] In the training method of the above-mentioned image style transfer model, a sample image is obtained; the sample image includes at least a first sample image and a second sample image, and the first sample image is obtained by downsampling the second sample image; then a first target style image corresponding to the first sample image is obtained, and a first neural network model is trained according to the first sample image and the first target style image to obtain a first image style transfer model; then a second target style image corresponding to the second sample image is obtained according to the first image style transfer model; finally, a second neural network model is trained according to the second sample image and the second target style image to obtain a target image style transfer model. In this way, in the process of gradually increasing the image resolution of the sample image, the image style transfer model can fit the image details well through step-by-step learning, thereby obtaining a more realistic detail processing result, so that the style image output by the target image style transfer model obtained by the final training has a stronger sense of reality, thereby improving the sense of reality of the obtained high-resolution style image, and avoiding the defect that the low-resolution style transfer relationship is difficult to adapt to the high resolution, resulting in a weaker sense of reality of the obtained high-resolution style image.

[0091] In an exemplary embodiment, in step S230, according to the first image style transfer model, a second target style image corresponding to the second sample image is obtained, which specifically includes: inputting the second sample image into the first image style transfer model for image style transfer processing to obtain a second target style image corresponding to the second sample image.

[0092] Specifically, the terminal inputs the second sample image into the first image style transfer model, performs image style transfer processing on the second sample image through the first image style transfer model, and obtains a second target style image corresponding to the second sample image.

[0093] Furthermore, if Figure 3 As shown, in step S240, the second neural network model is trained according to the second sample image and the second target style image to obtain a target image style transfer model, which can be specifically achieved by the following steps:

[0094] In step S310, the second sample image is input into the second neural network model for image style transfer processing to obtain a second predicted style image corresponding to the second sample image.

[0095] In step S320 , a loss value is obtained according to the difference between the second target style image and the second predicted style image.

[0096] In step S330, the second neural network model is trained according to the loss value to obtain a target image style transfer model.

[0097] The second predicted style image refers to a high-resolution predicted style image corresponding to the second sample image, such as a high-resolution face image after hair dyeing.

[0098] Specifically, the terminal inputs the second sample image into the second neural network model, performs image style transfer processing on the second sample image through the second neural network model, and obtains a second predicted style image corresponding to the second sample image; performs difference processing on the second target style image and the second predicted style image to obtain the difference between the second target style image and the second predicted style image, and combines the loss function to obtain the loss value of the second neural network model; if the loss value is greater than or equal to a preset threshold, the model parameters of the second neural network model are adjusted according to the loss value to obtain the second neural network model after the model parameters are adjusted; and repeats steps S310 to S330 to repeatedly train the second neural network model after the model parameters are adjusted, until the loss value obtained according to the trained second neural network model is less than the preset threshold, and the trained second neural network model is used as the target image style transfer model.

[0099] In an actual scenario, the terminal performs color conversion on the dyed hair area in the high-resolution face image through the trained low-resolution image style transfer model for color conversion of the hair area in the face image, and obtains a high-resolution face image after real hair dyeing. That is to say, in this way, a large number of high-resolution face images after real hair dyeing can be quickly produced, which solves the problem that high-resolution face images after real hair dyeing are difficult to obtain; then, the terminal inputs the high-resolution face image into a second neural network model, such as a deep neural network model, to obtain a predicted high-resolution face image after hair dyeing; based on the high-resolution face image after the real hair dyeing and the predicted high-resolution face image after hair dyeing, the model parameters in the second neural network model are optimized based on the stochastic gradient descent method to reduce the error between the predicted high-resolution face image after hair dyeing and the real high-resolution face image after hair dyeing; the above process is repeated, and after a preset number of iterations, the iteration is terminated, and the final second neural network model is used as the target image style transfer model.

[0100] The technical solution provided by the disclosed embodiment trains the second neural network model by a loss value obtained based on the difference between the second target style image and the second predicted style image, so that the trained target image style transfer model can well fit the image details of the high-resolution image, thereby obtaining a more realistic detail processing result, and further improving the accuracy of the high-resolution style image output by the target image style transfer model.

[0101] In an exemplary embodiment, the sample image also includes N third sample images, which are obtained by downsampling the second sample image, or the N third sample images are obtained by upsampling the first sample image; the image resolutions of the first sample image, the N third sample images, and the second sample image increase sequentially; N is a positive integer.

[0102] The third sample image may be one or more. Downsampling refers to reducing the image resolution of an image, thereby reducing the image, such as bilinear difference. Upsampling refers to increasing the image resolution of an image, thereby enlarging the image.

[0103] Specifically, the terminal obtains a sample image with a high image resolution and uses it as the second sample image; according to a preset downsampling instruction, the second sample image is downsampled to obtain a second sample image after downsampling, and uses it as the third sample image. Alternatively, the terminal performs upsampling on the first sample image according to a preset upsampling instruction to obtain a first sample image after upsampling, and uses it as the third sample image.

[0104] Furthermore, the terminal can also obtain a sample image with high image resolution and use it as the second sample image; according to a preset downsampling instruction, downsample the second sample image to obtain a second sample image after downsampling, and use it as the third sample image; according to the same preset downsampling instruction, perform the same downsampling process on the third sample image to obtain a third sample image after downsampling, and use it as the first sample image.

[0105] For example, the terminal obtains a facial image containing hair with high image resolution, such as 4k resolution, as the second sample image A; down-samples the second sample image A through a bilinear interpolation method to reduce the image resolution of the second sample image A, and obtains a corresponding facial image containing hair with medium resolution, such as 2k resolution, as the third sample image B; performs the same down-sampling process on the third sample image B through a bilinear interpolation method to reduce the image resolution of the third sample image B, and obtains a corresponding low-resolution facial image containing hair, such as 1k resolution, as the first sample image C.

[0106] The technical solution provided by the embodiment of the present disclosure obtains a third sample image by downsampling the second sample image; and obtains the first sample image by performing the same downsampling process on the third sample image, thereby ensuring that in the subsequent training process of image style transfer models corresponding to various image resolutions, the image contents of the various sample images are the same, but the image resolutions are different, so that the high-resolution style image output by the target image style transfer model obtained by subsequent training is more accurate, further improving the realism of the obtained high-resolution style image.

[0107] In an exemplary embodiment, the above step S230, according to the first image style transfer model, obtains the second target style image corresponding to the second sample image, and also includes: in order from low to high image resolution, taking the first third sample image among the N third sample images as the current third sample image, inputting the first image style transfer model to perform image style transfer processing, and obtaining a third target style image corresponding to the current third sample image; according to the current third sample image and the third target style image corresponding to the current third sample image, training the corresponding current third neural network model among the N third neural network models to obtain the current third image style transfer model; taking the N third sample images as the current third sample image, and inputting the first image style transfer model to perform image style transfer processing, and obtaining a third target style image corresponding to the current third sample image; according to the current third sample image and the third target style image corresponding to the current third sample image, training the corresponding current third neural network model among the N third neural network models to obtain the current third image style transfer model; The next third sample image of the current third sample image in the image is re-used as the current third sample image, and is input into the current third image style transfer model for image style transfer processing to obtain a third target style image corresponding to the current third sample image; the corresponding current third neural network model in the N third neural network models is re-trained according to the current third sample image and the third target style image corresponding to the current third sample image, until the current third image style transfer model obtained is the Nth third image style transfer model; the second sample image is input into the Nth third image style transfer model for image style transfer processing to obtain a second target style image corresponding to the second sample image.

[0108] Specifically, the terminal takes the first third sample image among N third sample images as the current third sample image, and inputs it into the first image style transfer model in order of image resolution from low to high, and performs image style transfer processing on the current third sample image through the first image style transfer model to obtain a third target style image corresponding to the current third sample image; according to the current third sample image and the third target style image corresponding to the current third sample image, the terminal trains the corresponding current third neural network model (such as the first third neural network model) among the N third neural network models to obtain a trained third neural network model, and uses it as the current third image style transfer model. Next, the terminal uses the next third sample image of the current third sample image among the N third sample images as the current third sample image, inputs it into the current third image style transfer model, performs image style transfer processing on the current third sample image through the current third image style transfer model, obtains the third target style image corresponding to the current third sample image, and re-executes the training of the corresponding current third neural network model (such as the second third neural network model) among the N third neural network models according to the current third sample image and the third target style image corresponding to the current third sample image, until the current third image style transfer model obtained is the Nth third image style transfer model, then the training is terminated; the second sample image is input into the Nth third image style transfer model, and the image style transfer processing is performed on the second sample image through the Nth third image style transfer model, to obtain the second target style image corresponding to the second sample image.

[0109] For example, when N is 1, the terminal obtains a first target style image corresponding to the first sample image, trains a first neural network model based on the first sample image and the first target style image, and obtains a first image style transfer model (such as a low-resolution image style transfer model); inputs the third sample image into the first image style transfer model for image style transfer processing, and obtains a third target style image corresponding to the third sample image; trains a third neural network model based on the third sample image and the third target style image, and obtains a third image style transfer model (such as a medium-resolution image style transfer model); inputs the second sample image into the third image style transfer model for image style transfer processing, and obtains a second target style image corresponding to the second sample image; trains the second neural network model based on the second sample image and the second target style image, and obtains a target image style transfer model (such as a high-resolution image style transfer model).

[0110] Furthermore, according to the third sample image and the third target style image, the third neural network model is trained to obtain a third image style transfer model, specifically including: inputting the third sample image into the third neural network model, performing image style transfer processing on the third sample image through the third neural network model to obtain a third predicted style image corresponding to the third sample image, and obtaining a second loss value of the third neural network model according to the difference between the third target style image and the third predicted style image; according to the second loss value, adjusting the model parameters of the third neural network model to obtain the third image style transfer model.

[0111] For example, the terminal performs difference processing on the third target style image and the third predicted style image to obtain the difference between the third target style image and the third predicted style image, and combines the loss function to obtain a second loss value of the third neural network model; if the second loss value is greater than or equal to the second preset threshold, the model parameters of the third image neural network model are adjusted according to the second loss value to obtain the third neural network model after the model parameters are adjusted; and the above process is repeated to repeatedly train the third neural network model after the model parameters are adjusted until the second loss value obtained according to the trained third neural network model is less than the second preset threshold, and the trained third neural network model is used as the third image style transfer model.

[0112] In an actual scenario, the terminal performs color conversion on the dyed hair area in the medium-resolution face image through the trained low-resolution image style transfer model for color conversion of the hair area in the face image, and obtains a medium-resolution face image after real hair dyeing. That is to say, in this way, a large number of medium-resolution face images after real hair dyeing can be quickly created; then, the terminal inputs the medium-resolution face image into the third neural network model to obtain a predicted medium-resolution face image after hair dyeing; based on the medium-resolution face image after the real hair dyeing and the predicted medium-resolution face image after hair dyeing, the model parameters in the third neural network model are optimized based on the stochastic gradient descent method to reduce the error between the predicted medium-resolution face image after hair dyeing and the real medium-resolution face image after hair dyeing; the above process is repeated, and after a preset number of iterations, the iteration is terminated, and the third neural network model finally obtained is used as the third image style transfer model.

[0113] According to the technical solution provided by the disclosed embodiments, in the process of gradually increasing the image resolution of the sample image, the image style transfer model can well fit the image details through step-by-step learning, thereby obtaining a more realistic detail processing result, so that the style image output by the target image style transfer model finally trained has a stronger sense of reality, thereby improving the realism of the obtained high-resolution style image, and avoiding the defect that the low-resolution style transfer relationship is difficult to adapt to the high resolution, resulting in the obtained high-resolution style image having a weaker sense of reality.

[0114] In an exemplary embodiment, the current third image style transfer model is trained in the following manner: from the third target style image corresponding to the current third sample image, a third target style image whose similarity with the image content of the current third sample image is greater than a first preset similarity and whose target conversion style is the same as that of the previous third image style transfer model of the current third image style transfer model is screened out as the third target style image to be referenced; from the current third sample image, a third sample image corresponding to the third target style image to be referenced is screened out as the third target sample image; according to the third target sample image and the third target style image corresponding to the third target sample image, the corresponding current third neural network model among the N third neural network models is trained to obtain the current third image style transfer model.

[0115] Among them, the third target style image to be referenced refers to a third target style image with successful image style transfer, such as a medium-resolution face image with successful hair color change, a medium-resolution face image with successful skin color change, etc., which can be obtained by manual screening or machine screening.

[0116] Specifically, considering that the third target style image is obtained by performing image style transfer on the third sample image of medium image resolution through the training of the first image style transfer model suitable for low-resolution image style transfer, or through the third image style transfer model of the previous image resolution of the current third image style transfer model, it is inevitable that there are images in which the image style transfer fails. Therefore, the terminal screens out the third target style image corresponding to the current third sample image, whose similarity with the image content of the current third sample image is greater than the first preset similarity, and whose target conversion style is the same as that of the previous third image style transfer model of the current third image style transfer model, from the third target style image corresponding to the current third sample image, and uses it as the third target style image to be referenced; then, the terminal screens out the third sample image whose corresponding third target style image is the third sample image to be referenced from the current third sample image, and uses it as the third target sample image; finally, the terminal trains the corresponding current third neural network model among the N third neural network models for multiple times according to the third target sample image and the third target style image corresponding to the third target sample image, until the number of training times reaches the preset number, then stops the training, and uses the third neural network model finally trained as the current third image style transfer model.

[0117] The technical solution provided by the disclosed embodiment screens out a third target sample image whose corresponding third target style image is a third target style image to be referenced from the current third sample image, and trains the current third neural network model multiple times based on the third target sample image and the third target style image corresponding to the third target sample image, which is beneficial to improving the accuracy of the second target style image corresponding to the second sample image output by the third image style transfer model obtained through training, and further improves the accuracy of the high-resolution reference image after style transfer.

[0118] In an exemplary embodiment, after the second sample image is input into the Nth third image style transfer model for image style transfer processing to obtain the second target style image corresponding to the second sample image, it also includes: from the second target style image, a second target style image having a similarity with the image content of the second sample image greater than a second preset similarity and having the same target conversion style as the Nth third image style transfer model is screened out as the second target style image to be referenced. Then, the above step S240, based on the second sample image and the second target style image, the second neural network model is trained to obtain the target image style transfer model, and it also includes: from the second sample image, a second sample image corresponding to the second target style image to be referenced is screened out as the second target sample image; based on the second target sample image and the second target style image corresponding to the second target sample image, the second neural network model is trained to obtain the target image style transfer model.

[0119] Among them, the second target style image to be referenced refers to the second target style image with successful image style transfer, such as a high-resolution face image with successful hair color change, a high-resolution face image with successful skin color change, etc., which can be obtained by manual screening or machine screening.

[0120] Specifically, considering that the second target style image is obtained by performing image style transfer on the second sample image with high image resolution through training of a third image style transfer model (such as the Nth third image style transfer model) suitable for medium resolution image style transfer, it is inevitable that there will be images in which the image style transfer fails. Therefore, the terminal screens out from the second target style image a second target style image whose image content similarity with the second sample image is greater than a second preset similarity and has the same target transfer style as the Nth third image style transfer model, and uses it as the second target style image to be referenced; then, the terminal screens out from the second sample image a second sample image whose corresponding second target style image is the second target style image to be referenced, and uses it as the second target sample image; finally, the terminal trains the second neural network model for multiple times according to the second target sample image and the second target style image corresponding to the second target sample image, until the number of training times reaches the preset number, then stops the training, and uses the second neural network model finally trained as the target image style transfer model.

[0121] The technical solution provided by the embodiment of the present disclosure screens out, from the second sample image, a second target sample image whose corresponding second target style image is the second target style image to be referenced, and trains the second neural network model multiple times according to the second target sample image and the second target style image corresponding to the second target sample image, which is beneficial to improving the accuracy of the target style image with high image resolution output by the target image style transfer model obtained through training, further improving the accuracy of the obtained high-resolution style image, and at the same time achieving the purpose of training a high-resolution style transfer model in the absence of high-resolution paired images.

[0122] Figure 4 is a flowchart of an image style transfer method according to an exemplary embodiment. Figure 2 As shown, the image style transfer method is used as Figure 1 In the terminal shown, the following steps are included:

[0123] In step S410, an image to be processed is obtained.

[0124] The image to be processed refers to a high-resolution image, such as a high-resolution face image.

[0125] In step S420, the image to be processed is input into the target image style transfer model to obtain a target style image corresponding to the image to be processed.

[0126] The target image style transfer model is obtained according to the training method of the image style transfer model described in any of the above embodiments.

[0127] For example, the terminal obtains a high-resolution facial image (including hair), and inputs the high-resolution facial image into a trained high-resolution image style transfer model for performing color conversion on the hair area in the facial image. Through the high-resolution image style transfer model, the color of the dyed hair area in the high-resolution facial image is converted to obtain a high-resolution facial image after dyeing, such as a high-resolution yellow hair image.

[0128] In the above-mentioned image style transfer method, a target style image corresponding to the image to be processed is obtained by training a target image style transfer model, which is beneficial to improving the realism of the obtained high-resolution style image and avoiding the defect that the low-resolution style transfer relationship is difficult to adapt to the high resolution, resulting in a weak sense of reality of the obtained high-resolution style image.

[0129] In an exemplary embodiment, a high-resolution texture style transfer method based on model self-learning is also provided. The method can be applied to the hair color conversion scene of a portrait. The hair area needs to be converted in color, while other areas need to remain unchanged. The principle is as follows: Figure 5 As shown in FIG. 1 , this method uses low-resolution paired data to train a low-resolution neural network, and then uses the neural network to create a high-resolution paired graph; after manual screening, the created high-resolution paired graph is used to train a high-resolution neural network. The specific process is as follows:

[0130] Dataset preparation:

[0131] 1) Prepare a large number of high-resolution portrait images (including hair), such as 4k resolution, to form dataset A hi ; These pictures are collected from mobile phones, cameras, and downloaded from public pictures on the Internet.

[0132] 2) High-resolution image A hi The image resolution is reduced by downsampling methods (such as bilinear interpolation) to obtain the corresponding medium-resolution dataset A mi .

[0133] 3) Take the medium resolution image A mi The image resolution is reduced by downsampling methods (such as bilinear interpolation) to obtain the corresponding low-resolution dataset A li .

[0134] 4) Use manual or AI methods to convert each low-resolution image A li Transformed into the corresponding yellow hair picture B li For example, designers can li Manually edit the photo to get the hair image with corresponding color conversion.

[0135] 5) The above data were sorted to obtain each high-resolution image A hi Corresponding reduced resolution images A at different resolutions mi and A li , and the corresponding low-resolution yellow hair image B li .

[0136] Training low-resolution AI models:

[0137] 1) Prepare a dataset D according to the above process. The dataset contains several image pairs A li and B li (i=1,2,3,4…), picture A li For pictures taken by mobile phones, cameras and other devices (such as original portrait photos), the corresponding stylized pictures B li (such as stylized portrait images). Figure 5 For example, Ai is the original portrait, and Bi is the picture after hair dyeing.

[0138] 2) Prepare three deep neural networks G l , Gm , G h , which are used to complete the style conversion of three different resolutions: low, medium and high.

[0139] 3) From the dataset, select a low-resolution image A li Paired style-transformed image B li .

[0140] 4) A li Feed G l Get B li ', using stochastic gradient descent to optimize G l Parameters in, reduce B li 'With B li of error.

[0141] 5) Repeat 3) to 4), iterate N l After times, the iteration is terminated.

[0142] 6) Set all A mi Feed G l Get the corresponding output B mi , as a medium-resolution reference image of the hair after dyeing.

[0143] Training a medium-resolution AI model:

[0144] 7) From the dataset, select any medium-resolution image A mi Paired style-transformed image B mi .

[0145] 8) A mi Feed G m Get B mi ', using stochastic gradient descent to optimize G m Parameters in, reduce B mi 'With B mi of error.

[0146] 9) Repeat 7) to 8) for N iterations l After times, the iteration is terminated.

[0147] 10) Set all A hi Feed G m Get the corresponding output B hi , as a high-resolution reference image of the hair after dyeing.

[0148] Training high-resolution AI models:

[0149] 11) From the dataset, select a high-resolution image A hi Paired style-transformed image B hi .

[0150] 12) A hi Feed G h Get B hi ', using stochastic gradient descent to optimize G h Parameters in, reduce B hi 'With B hi of error.

[0151] 13) Repeat 11) to 12) for N iterations l After times, the iteration is terminated.

[0152] Model deployment:

[0153] 14) Then G h Networks are embedded in the image processing pipeline.

[0154] 15) In the usage scenario, the pictures collected from mobile phones, cloud terminals, etc. are sent to G as the input of the neural network. h , and use the network output image instead of the original image as the output.

[0155] The above-mentioned high-resolution texture style transfer method based on model self-learning can achieve the following technical effects: (1) Improved conversion details: The neural network has the ability to generate details. In the process of gradually increasing the resolution, the neural network fits the picture details well through step-by-step learning, and can obtain more realistic detail processing results; (2) Large-rate resolution improvement: By gradually improving the resolution, the resolution is gradually improved through multiple iterations; each time the resolution is improved to a level close to the original resolution; after multiple layers of resolution improvement, through multiple iterations, the overall resolution is greatly improved, and the high-resolution conversion model can be trained in the absence of high-resolution paired pictures.

[0156] It should be understood that although Figure 2-Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 4 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0157] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0158] Figure 6 is a block diagram of a training device for an image style transfer model according to an exemplary embodiment. Figure 6 The device includes a first acquisition unit 610, a first training unit 620, a second acquisition unit 630 and a second training unit 640.

[0159] The first acquisition unit 610 is configured to acquire a sample image; the sample image includes at least a first sample image and a second sample image, and the first sample image is obtained by downsampling the second sample image.

[0160] The first training unit 620 is configured to execute acquiring a first target style image corresponding to the first sample image, and to train a first neural network model according to the first sample image and the first target style image to obtain a first image style transfer model.

[0161] The second acquisition unit 630 is configured to execute the first image style transfer model to obtain a second target style image corresponding to the second sample image.

[0162] The second training unit 640 is configured to train the second neural network model according to the second sample image and the second target style image to obtain a target image style transfer model.

[0163] In an exemplary embodiment, the second acquisition unit 630 is further configured to input the second sample image into the first image style transfer model to perform image style transfer processing to obtain a second target style image corresponding to the second sample image;

[0164] The second training unit 640 is further configured to input the second sample image into the second neural network model for image style transfer processing to obtain a second predicted style image corresponding to the second sample image; obtain a loss value based on the difference between the second target style image and the second predicted style image; and train the second neural network model based on the loss value to obtain a target image style transfer model.

[0165] In an exemplary embodiment, the sample images further include N third sample images, which are obtained by downsampling the second sample images, or by upsampling the first sample images; the image resolutions of the first sample image, the N third sample images, and the second sample image are increased in sequence; N is a positive integer;

[0166] The second acquisition unit 630 is further configured to execute, in order of image resolution from low to high, taking the first third sample image among the N third sample images as the current third sample image, inputting the first image style transfer model to perform image style transfer processing, and obtaining a third target style image corresponding to the current third sample image; training the corresponding current third neural network model among the N third neural network models according to the current third sample image and the third target style image corresponding to the current third sample image, and obtaining a current third image style transfer model; taking the next third sample image of the current third sample image among the N third sample images as the current third sample image again, inputting the current third image style transfer model to perform image style transfer processing, and obtaining a third target style image corresponding to the current third sample image; retraining the corresponding current third neural network model among the N third neural network models according to the current third sample image and the third target style image corresponding to the current third sample image, until the current third image style transfer model obtained is the Nth third image style transfer model; inputting the second sample image into the Nth third image style transfer model to perform image style transfer processing, and obtaining a second target style image corresponding to the second sample image.

[0167] In an exemplary embodiment, the second acquisition unit 630 is further configured to perform the following steps: screening out, from the third target style image corresponding to the current third sample image, a third target style image whose similarity with the image content of the current third sample image is greater than a first preset similarity and whose target conversion style is the same as that of the previous third image style transfer model of the current third image style transfer model, as the third target style image to be referenced; screening out, from the current third sample image, a third sample image corresponding to the third target style image to be referenced as the third target sample image; and training the corresponding current third neural network model among the N third neural network models according to the third target sample image and the third target style image corresponding to the third target sample image to obtain the current third image style transfer model.

[0168] In an exemplary embodiment, the training device of the image style transfer model further includes an image screening unit configured to perform screening, from the second target style image, a second target style image having a similarity with the image content of the second sample image greater than a second preset similarity and having the same target conversion style as the Nth third image style transfer model, as the second target style image to be referenced;

[0169] The second training unit 640 is further configured to perform the operation of screening out a second sample image corresponding to a second target style image to be referenced from the second sample image as a second target sample image; and training a second neural network model according to the second target sample image and the second target style image corresponding to the second target sample image to obtain a target image style transfer model.

[0170] Figure 7 is a block diagram of an image style transfer device according to an exemplary embodiment. Figure 7 , the device includes an acquisition unit 710 and a processing unit 720.

[0171] The acquisition unit 710 is configured to acquire the image to be processed.

[0172] The processing unit 720 is configured to input the image to be processed into the target image style transfer model to obtain a target style image corresponding to the image to be processed; the target image style transfer model is obtained according to the training method of the image style transfer model.

[0173] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0174] Figure 8 1 is a block diagram of an electronic device 800 for performing a training method of an image style transfer model or an image style transfer method according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0175] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0176] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0177] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, an optical disk, or a graphene memory.

[0178] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0179] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0180] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory Z04 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting an audio signal.

[0181] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0182] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or the components of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the device 800 and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0183] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0184] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0185] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the above instructions can be executed by the processor 820 of the electronic device 800 to complete the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0186] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by the processor 820 of the electronic device 800 to complete the above method.

[0187] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. may also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.

[0188] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0189] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an image style transfer model, characterized in that: include: Get a sample image; The sample images at least include a first sample image and a second sample image, and the first sample image is obtained by downsampling the second sample image; Acquire a first target style image corresponding to the first sample image, and train a first neural network model according to the first sample image and the first target style image to obtain a first image style transfer model; According to the first image style transfer model, a second target style image corresponding to the second sample image is obtained; the second target style image is obtained by performing image style transfer processing on the second sample image by the first image style transfer model; According to the second sample image and the second target style image, a second neural network model is trained to obtain a target image style transfer model; the target image style transfer model is obtained by adjusting the model parameters of the second neural network model through the difference between the second target style image and the second predicted style image corresponding to the second sample image; the second predicted style image is obtained by performing image style transfer processing on the second sample image by the second neural network model.

2. The method for training an image style transfer model according to claim 1, characterized in that: The obtaining, according to the first image style transfer model, a second target style image corresponding to the second sample image includes: Inputting the second sample image into the first image style transfer model to perform image style transfer processing to obtain a second target style image corresponding to the second sample image; The step of training a second neural network model according to the second sample image and the second target style image to obtain a target image style transfer model includes: Inputting the second sample image into a second neural network model to perform image style transfer processing to obtain a second predicted style image corresponding to the second sample image; Obtaining a loss value according to a difference between the second target style image and the second predicted style image; The second neural network model is trained according to the loss value to obtain the target image style transfer model.

3. The training method of the image style transfer model according to claim 1, characterized in that: The sample images further include N third sample images, where the N third sample images are obtained by downsampling the second sample images, or the N third sample images are obtained by upsampling the first sample images; The image resolutions of the first sample image, the N third sample images and the second sample image are increased in sequence; N is a positive integer; The step of obtaining a second target style image corresponding to the second sample image according to the first image style transfer model further includes: In order of image resolution from low to high, taking the first third sample image among the N third sample images as the current third sample image, inputting the first image style transfer model to perform image style transfer processing, and obtaining a third target style image corresponding to the current third sample image; According to the current third sample image and the third target style image corresponding to the current third sample image, training a corresponding current third neural network model among the N third neural network models to obtain a current third image style transfer model; Taking the next third sample image of the current third sample image among the N third sample images as the current third sample image, inputting the current third image style transfer model into the current third image to perform image style transfer processing, and obtaining a third target style image corresponding to the current third sample image; Retraining the corresponding current third neural network model among the N third neural network models according to the current third sample image and the third target style image corresponding to the current third sample image, until the obtained current third image style transfer model is the Nth third image style transfer model; The second sample image is input into the Nth third image style transfer model to perform image style transfer processing to obtain a second target style image corresponding to the second sample image.

4. The method for training an image style transfer model according to claim 3, characterized in that: After inputting the second sample image into the Nth third image style transfer model for image style transfer processing to obtain a second target style image corresponding to the second sample image, the method further includes: From the second target style image, a second target style image having an image content similarity with the second sample image greater than a second preset similarity and having the same target conversion style as the Nth third image style transfer model is selected as the second target style image to be referenced; The step of training a second neural network model according to the second sample image and the second target style image to obtain a target image style transfer model further includes: Filtering out, from the second sample images, a second sample image corresponding to the second target style image to be referenced as a second target sample image; The second neural network model is trained according to the second target sample image and the second target style image corresponding to the second target sample image to obtain the target image style transfer model.

5. The method for training an image style transfer model according to claim 3, characterized in that: After inputting the second sample image into the Nth third image style transfer model for image style transfer processing to obtain a second target style image corresponding to the second sample image, the method further includes: From the second target style image, a second target style image having an image content similarity with the second sample image greater than a second preset similarity and having the same target conversion style as the Nth third image style transfer model is selected as the second target style image to be referenced; The step of training a second neural network model according to the second sample image and the second target style image to obtain a target image style transfer model further includes: Filtering out, from the second sample images, a second sample image corresponding to the second target style image to be referenced as a second target sample image; The second neural network model is trained according to the second target sample image and the second target style image corresponding to the second target sample image to obtain the target image style transfer model.

6. A method for image style transfer, characterized in that: include: Get the image to be processed; Inputting the image to be processed into a target image style transfer model to obtain a target style image corresponding to the image to be processed; The target image style transfer model is obtained according to the training method of the image style transfer model according to any one of claims 1 to 5.

7. A training device for an image style transfer model, characterized in that: include: A first acquisition unit is configured to acquire a sample image; The sample images at least include a first sample image and a second sample image, and the first sample image is obtained by downsampling the second sample image; A first training unit is configured to obtain a first target style image corresponding to the first sample image, and train a first neural network model according to the first sample image and the first target style image to obtain a first image style transfer model; A second acquisition unit is configured to execute the first image style transfer model to obtain a second target style image corresponding to the second sample image; The second target style image is obtained by performing image style transfer processing on the second sample image by the first image style transfer model; The second training unit is configured to execute training of the second neural network model according to the second sample image and the second target style image to obtain a target image style transfer model; the target image style transfer model is obtained by adjusting the model parameters of the second neural network model through the difference between the second target style image and the second predicted style image corresponding to the second sample image; the second predicted style image is obtained by performing image style transfer processing on the second sample image by the second neural network model.

8. The training device for the image style transfer model according to claim 7, characterized in that: The second acquisition unit is further configured to input the second sample image into the first image style transfer model to perform image style transfer processing to obtain a second target style image corresponding to the second sample image; The second training unit is further configured to input the second sample image into a second neural network model to perform image style transfer processing to obtain a second predicted style image corresponding to the second sample image; Obtaining a loss value according to a difference between the second target style image and the second predicted style image; The second neural network model is trained according to the loss value to obtain the target image style transfer model.

9. The training device for the image style transfer model according to claim 7, characterized in that: The sample images further include N third sample images, where the N third sample images are obtained by downsampling the second sample images, or the N third sample images are obtained by upsampling the first sample images; The image resolutions of the first sample image, the N third sample images and the second sample image are increased in sequence; N is a positive integer; The second acquisition unit is further configured to perform, in order of image resolution from low to high, taking the first third sample image among the N third sample images as the current third sample image, inputting the first image style transfer model to perform image style transfer processing, and obtaining a third target style image corresponding to the current third sample image; According to the current third sample image and the third target style image corresponding to the current third sample image, the corresponding current third neural network model in the N third neural network models is trained to obtain a current third image style transfer model; the next third sample image of the current third sample image in the N third sample images is used as the current third sample image again, and input into the current third image style transfer model for image style transfer processing to obtain the third target style image corresponding to the current third sample image; according to the current third sample image and the third target style image corresponding to the current third sample image, the corresponding current third neural network model in the N third neural network models is retrained until the current third image style transfer model obtained is the Nth third image style transfer model; the second sample image is input into the Nth third image style transfer model for image style transfer processing to obtain a second target style image corresponding to the second sample image.

10. The training device for the image style transfer model according to claim 9, characterized in that: The second acquisition unit is further configured to perform, from the third target style image corresponding to the current third sample image, screening out a third target style image having an image content similarity with the current third sample image greater than a first preset similarity and having the same target conversion style as a previous third image style transfer model of the current third image style transfer model, as the third target style image to be referenced; Filtering out, from the current third sample images, a third sample image corresponding to the third target style image to be referenced as a third target sample image; According to the third target sample image and the third target style image corresponding to the third target sample image, the corresponding current third neural network model among the N third neural network models is trained to obtain the current third image style transfer model.

11. The training device for an image style transfer model according to claim 9, characterized in that: The device also includes an image screening unit configured to screen out, from the second target style image, a second target style image having an image content similarity with the second sample image greater than a second preset similarity and having the same target conversion style as the Nth third image style transfer model, as a second target style image to be referenced; The second training unit is further configured to screen out, from the second sample images, a second sample image corresponding to the second target style image to be referenced as a second target sample image; The second neural network model is trained according to the second target sample image and the second target style image corresponding to the second target sample image to obtain the target image style transfer model.

12. An image style transfer device, characterized in that: include: An acquisition unit, configured to acquire an image to be processed; A processing unit is configured to input the image to be processed into a target image style transfer model to obtain a target style image corresponding to the image to be processed; The target image style transfer model is obtained according to the training method of the image style transfer model according to any one of claims 1 to 5.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 6.

15. A computer program product, comprising instructions, characterized in that: When the instructions are executed by a processor of an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Image style migration method and device and computer readable storage medium

    CN109919829A

  • Training method of image style migration model and image style migration method and device

    CN112581361A

  • Image style migration model training method, image style migration method and device

    CN112734627A

  • Training set acquisition method and apparatus, and electronic device

    CN113269213A

  • Image style migration method and device, electronic device and storage medium

    CN113436062A