Image style transfer model training method, device and electronic equipment

Abstract

The present application provides a training method, device and electronic device for an image style transfer model, the method comprising: obtaining an original image and a real image; inputting the original image into a generator of the image style transfer model to obtain a stylized image; extracting a first edge region of a target object in the stylized image and a second edge region of the target object in the real image, training a generator and a discriminator of the image style transfer model based on the first edge region and the second edge region, and using the trained generator as the image style transfer model. In the present application, the training process of the model can only use the edge region as input, so that the model can only process the features of the edge region, reducing the amount of calculation. In addition, the training process can focus more on the fusion of edge regions in the image style transfer process, so that the changes in the detailed texture of the edge region can be easily captured, thereby improving the edge region repair effect of the image stylization result.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of training image style transfer models, and in particular, to a method and device for training an image style transfer model, a method and device for generating a stylized image, an electronic device, a computer storage medium, and a computer program product. Background Art

[0002] Image stylization tasks can provide users with a variety of stylized images. For example, in a portrait hair dyeing task, the hair area needs to be converted in color, while other areas can remain unchanged. This type of task can segment different areas, perform style conversion on each area, and then splice them together. The fusion effect of the splicing position affects the quality of the stylized image.

[0003] In the related technology, the generator in the Generative Adversarial Networks (GAN) model can be used to perform stylized processing to obtain a stylized image, and the edge position of the style area in the stylized image is marked. Then, the discriminator of the GAN model is used to identify the edge fusion effect of the style area in the stylized image. Through game-style training between the generator and the discriminator, the effect of the stylized processing provided by the generator can be improved.

[0004] However, in the current solution, the discriminator needs to process the entire image, resulting in excessive consumption of computing resources. In addition, since the area where the edge position is located in the entire image accounts for too small a proportion, the discriminator is difficult to capture the difference in edge position, making training more difficult. Summary of the invention

[0005] The embodiments of the present application provide a training method and device for an image style transfer model, a method and device for generating a stylized image, an electronic device, a computer storage medium, and a computer program product to solve the problem that the discriminator in the related art needs to process the entire image, resulting in excessive consumption of computing resources. In addition, since the area where the edge position is located in the entire image accounts for too small a proportion, it is difficult for the discriminator to capture the difference in the edge position, resulting in greater training difficulty.

[0006] In a first aspect, an embodiment of the present application provides a method for training an image style transfer model, the method comprising:

[0007] Acquire an original image and a real image, wherein both the original image and the real image contain a target object, and a style attribute of the target object contained in the original image is different from a style attribute of the target object contained in the real image;

[0008] Inputting the original image into a generator of a neural network model to obtain a stylized image, wherein the stylized image is an image obtained by stylizing a target object in the original image; the neural network model further includes a discriminator;

[0009] Extracting a first edge region of the target object in the stylized image and a second edge region of the target object in the real image;

[0010] The generator and the discriminator are trained according to the first edge region and the second edge region, and the trained generator is used as an image style transfer model.

[0011] In an optional implementation, the training of the generator and the discriminator according to the first edge region and the second edge region, and using the trained generator as an image style transfer model includes:

[0012] Performing image segmentation on the first edge area to obtain a plurality of first image blocks of preset sizes;

[0013] Performing image segmentation on the second edge area to obtain a plurality of second image blocks of the preset size;

[0014] The generator and the discriminator are trained according to the first image block and the second image block, and the trained generator is used as the image style transfer model.

[0015] In an optional implementation, the training of the generator and the discriminator according to the first image block and the second image block includes:

[0016] Training the discriminator according to the first image block and the second image block;

[0017] The generator is trained according to the first image block and the discriminator.

[0018] In an optional implementation, the training of the discriminator according to the first image block and the second image block includes:

[0019] Inputting the first image block and the second image block into the discriminator respectively, obtaining a first determination result corresponding to the first image block and a second determination result corresponding to the second image block;

[0020] Determine a first loss value according to a difference between a first true value of the first image block and the first determination result, and determine a second loss value according to a difference between a true value of the second image block and the second determination result; the first true value reflects a value taken when a determination result of the discriminator on the input object is false; the true value of the second image block reflects a value taken when a determination result of the discriminator on the input object is true;

[0021] The discriminator is trained alternately according to the first loss value and the second loss value.

[0022] In an optional implementation, the step of training the generator according to the first image block and the discriminator includes:

[0023] Inputting the first image block into the discriminator to obtain a third determination result corresponding to the first image block;

[0024] Determine a third loss value according to a difference between a second true value of the first image block and the third determination result; the second true value reflects a value when the determination result of the discriminator on the input object is true;

[0025] The generator is trained according to the third loss value.

[0026] In an optional implementation, the training of the generator and the discriminator according to the first edge region and the second edge region, and using the trained generator as an image style transfer model includes:

[0027] After performing a preset number of iterative training operations on the generator and the discriminator, the training is terminated, and the trained generator is used as an image style transfer model. One iterative training operation includes training the generator once and training the discriminator once.

[0028] In a second aspect, an embodiment of the present application provides a method for generating a stylized image, the method comprising:

[0029] Inputting the image to be processed into an image style transfer model to obtain a stylized image output by the image style transfer model;

[0030] The image style transfer model is obtained by training according to the training method of the image style transfer model.

[0031] In a third aspect, an embodiment of the present application provides a training device for an image style transfer model, the device comprising:

[0032] an acquisition module, configured to acquire an original image and a real image, wherein both the original image and the real image contain a target object, and a style attribute of the target object contained in the original image is different from a style attribute of the target object contained in the real image;

[0033] A first style module is configured to input the original image into a generator of a neural network model to obtain a stylized image, wherein the stylized image is an image obtained by stylizing a target object in the original image; the neural network model further includes a discriminator;

[0034] a segmentation module, configured to extract a first edge region of the target object in the stylized image and a second edge region of the target object in the real image;

[0035] A training module is configured to train the generator and the discriminator according to the first edge region and the second edge region, and use the trained generator as an image style transfer model.

[0036] In an optional implementation, the training module includes:

[0037] A first segmentation submodule is configured to perform image segmentation on the first edge area to obtain a plurality of first image blocks of preset sizes;

[0038] A second segmentation submodule is configured to perform image segmentation on the second edge area to obtain a plurality of second image blocks of the preset size;

[0039] A training submodule is configured to train the generator and the discriminator according to the first image block and the second image block, and use the trained generator as the image style transfer model.

[0040] In an optional implementation, the training submodule includes:

[0041] A first training unit is configured to train the discriminator according to the first image block and the second image block;

[0042] The second training unit is configured to train the generator according to the first image block and the discriminator.

[0043] In an optional implementation, the first training unit includes:

[0044] A first training subunit is configured to input the first image block and the second image block into the discriminator respectively, to obtain a first determination result corresponding to the first image block and a second determination result corresponding to the second image block;

[0045] a second training subunit, configured to determine a first loss value according to a difference between a first true value of the first image block and the first determination result, and to determine a second loss value according to a difference between a true value of the second image block and the second determination result; the first true value reflects a value taken when a determination result of the discriminator on the input object is false; and the true value of the second image block reflects a value taken when a determination result of the discriminator on the input object is true;

[0046] The alternating training subunit is configured to alternately train the discriminator according to the first loss value and the second loss value.

[0047] In an optional implementation manner, the second training unit includes:

[0048] a third training subunit, configured to input the first image block into the discriminator to obtain a third determination result corresponding to the first image block;

[0049] a fourth training subunit, configured to determine a third loss value according to a difference between a second true value of the first image block and the third determination result; the second true value reflects a value taken when the determination result of the discriminator on the input object is true;

[0050] A training subunit is configured to train the generator according to the third loss value.

[0051] In an optional implementation, the training module includes:

[0052] The iterative training submodule is configured to terminate the training after performing a preset number of iterative training operations on the generator and the discriminator, and use the trained generator as the image style transfer model. One iterative training operation includes training the generator once and training the discriminator once.

[0053] In a fourth aspect, an embodiment of the present application provides a stylized image generation device, the device comprising:

[0054] A style processing module is configured to input the image to be processed into an image style transfer model to obtain a stylized image output by the image style transfer model;

[0055] Wherein, the image style transfer model is obtained according to a training device of the image style transfer model.

[0056] In a fifth aspect, an embodiment of the present application further provides an electronic device comprising a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the training of the image style transfer model and the stylized image generation method.

[0057] In a sixth aspect, an embodiment of the present application further provides a computer storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the training of the image style transfer model and the stylized image generation method.

[0058] In a seventh aspect, an embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the training of the image style transfer model and the stylized image generation method.

[0059] In an embodiment of the present application, a first edge region of a target object in a stylized image and a second edge region of a target object in a real image can be extracted. This allows the subsequent training process to use only the edge region as input, so that during the training process, the training process of the model can use only the edge region as input, so that the model can only process the features of the edge region, reducing the amount of calculation. In addition, the training process can focus more on the fusion of edge regions in the image style transfer process, so that the changes in the detailed texture of the edge region can be easily captured, thereby improving the edge region repair effect of the image stylization result.

[0060] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0062] Figure 1 It is a flowchart of the steps of a training method of an image style transfer model provided in an embodiment of the present application;

[0063] Figure 2 It is an architecture diagram of a generative adversarial network model provided in an embodiment of the present application;

[0064] Figure 3 is a stylized image provided by an embodiment of the present application;

[0065] Figure 4 is a flowchart of the steps of a stylized image generation method provided in an embodiment of the present application;

[0066] Figure 5It is a flowchart of the steps of another method for training an image style transfer model provided in an embodiment of the present application;

[0067] Figure 6 This is a schematic diagram of edge area segmentation provided in an embodiment of the present application;

[0068] Figure 7 It is a block diagram of a training device for an image style transfer model provided in an embodiment of the present application;

[0069] Figure 8 is a block diagram of a stylized image generation device provided in an embodiment of the present application;

[0070] Fig. 9 is a logic block diagram of an electronic device according to an embodiment of the present application;

[0071] Fig.10 It is a logic block diagram of an electronic device according to another embodiment of the present application. DETAILED DESCRIPTION

[0072] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0073] Figure 1 is a flowchart of the steps of a training method for an image style transfer model provided in an embodiment of the present application, such as Figure 1 As shown, the method includes:

[0074] Step 101, obtaining an original image and a real image.

[0075] Both the original image and the real image contain a target object, and a style attribute of the target object contained in the original image is different from a style attribute of the target object contained in the real image.

[0076] The training method of an image style transfer model provided in the embodiment of the present application can specifically refer to the training of a generative adversarial network model (GAN model), and refer to Figure 2 , which shows an architecture diagram of a generative adversarial network model provided in an embodiment of the present application. The GAN model includes a generator and a discriminator. The GAN model is a neural network model that can generate preferred outputs through mutual game learning between the generator and the discriminator.

[0077] Specifically, the training data for training the image style transfer model may include several real images and original images, wherein both the original images and the real images contain target objects, and the style attributes of the target objects contained in the original images are different from the style attributes of the target objects contained in the real images.

[0078] For example, for stylized processing of changing hair color to light yellow, multiple portraits of people with black hair can be prepared as original images, and multiple portraits of people with naturally light yellow hair can be prepared as real images.

[0079] Step 102: input the original image into the generator of the neural network model to obtain a stylized image, wherein the stylized image is an image obtained after stylizing the target object in the original image; the neural network model also includes a discriminator.

[0080] Furthermore, after preparing the training data, the original image can be first subjected to stylized processing by the generator in the GAN architecture to obtain a stylized image. For example, assuming that the generator is a model that provides a stylized processing for changing the hair color to light yellow hair color, the hair area in the stylized image obtained after the stylized processing is added with the light yellow hair color style feature.

[0081] Step 103: extract a first edge region of the target object in the stylized image and a second edge region of the target object in the real image.

[0082] In order to further improve the fusion effect of the style area and other areas in the stylized image, we can focus more on the edge area between the style area and other areas in the training process of the generator and discriminator under the GAN architecture, and reduce the interference of the area outside the edge area. Figure 1 , the embodiment of the present application can extract the first edge region of the target object in the stylized image, and the second edge region of the target object in the real image. The subsequent training process can only use the edge region as input, so that during the training process, the training process of the model can only use the edge region as input, so that the model can only process the features of the edge region, reducing the amount of calculation. In addition, the training process can focus more on the edge region fusion in the image style transfer process, so that the changes in the detailed texture of the edge region can be easily captured, improving the edge region repair effect of the image stylization result.

[0083] For example, refer to Figure 3, which shows a stylized image provided by an embodiment of the present application. Assuming that in the stylized image, region 10 is stylized, region 10 can be used as the style region of the target object, and a first edge region 30 (the region covered by the dotted line) at the junction between region 10 and an adjacent region 20 is extracted. The extraction of the second edge region of the real image is similar, wherein, if the style region of the target object is the hair region, the target region of the target object in the real image is also the hair region.

[0084] Step 104: train the generator and the discriminator according to the first edge region and the second edge region, and use the trained generator as an image style transfer model.

[0085] In the embodiment of the present application, since the parameters of the generator are not perfect in the early stage of training, the effect and quality of the stylized image processed by the generator are poor, which also makes the fusion effect of the first edge area in the stylized image poor. The real image is a pre-collected image that meets the ideal effect of the generator output, so the stylization effect of the target area corresponding to the style area in the real image is more ideal, and the fusion effect of the second edge area is better.

[0086] Therefore, in one training implementation, reference Figure 1 In an iterative operation, the first edge region and the second edge region can be used as inputs, and the discriminator can be trained to identify the first edge region as "fake" and the second edge region as "true". Specifically, the output results of the discriminator include "true" and "false". The "true" case means that the edge region is highly similar to the ideal edge region, and the "false" case means that the edge region is highly dissimilar to the ideal edge region. In the process of training the discriminator, the first edge region in the stylized image output by the generator can be considered as a virtual fake region with poor fusion quality, and the second edge region in the real image can be considered as a real real region with a natural fusion effect.

[0087] After training the parameters of the discriminator, the discriminator's ability to distinguish true from false edge areas may be improved. The generator needs to be further trained later. In the process of training the generator, the first edge area in the stylized image output by the generator needs to be able to deceive the discriminator, specifically, to be identified by the discriminator as a real area with a natural fusion effect. This makes the training goal of the generator: the first edge area in the stylized image output by the generator can be identified as "true" by the discriminator. Then, the first edge area in the stylized image obtained by the generator processing the original image can be input into the discriminator, and the loss value is calculated using the output value of the discriminator and the true value (at this time, the true value is that the discriminator can identify the first edge area as "true"), and then the loss value and loss function are used to train the parameters of the generator. After training, the fusion quality of the first edge area in the stylized image generated by the generator may be improved.

[0088] At the beginning of the next round of iteration, the parameters of the generator and the discriminator are optimized. After the next round of iteration, the parameters of the generator and the discriminator are further optimized until the preset number of iterations is reached. The iteration is terminated and the final generator is used as the image style transfer model.

[0089] It should be noted that in another training implementation method of the embodiment of the present application, the generator can be trained first and then the discriminator can be trained. The embodiment of the present application does not limit the actual training order.

[0090] The embodiment of the present application utilizes the mutual game learning between the generator and the discriminator in the GAN architecture to improve the model parameters of the two and produce high-quality output. After the image style transfer model is established, the image style transfer model can be embedded in the image processing process, such as mobile terminals, servers, and clouds, to provide high-quality stylized processing functions, so that users can use the image to be processed as the input of the image style transfer model, add stylized features to the image to be processed, and obtain a stylized image with high stylized quality and fusion quality.

[0091] Optionally, step 104 may specifically include:

[0092] Sub-step 1041, after performing a preset number of iterative training operations on the generator and the discriminator, terminate the training, and use the trained generator as an image style transfer model, wherein one iterative training operation includes training the generator once and training the discriminator once.

[0093] In an embodiment of the present application, each iterative training operation includes the operation of training the generator and the discriminator once respectively, so that the parameters of the generator and the discriminator are optimized. In order to achieve the purpose of training to obtain better model parameters, multiple rounds of iterative training operations are required to make the model parameters reach the expected value. After reaching a preset number of iterative training operations, it can be considered that the model parameters have reached the expected value, and the iteration is terminated, and the final generator is used as the image style transfer model.

[0094] In addition, the preset number of times that the iterative training operation finally reaches can be set according to actual needs. In one method, after reaching the preset number of times, the similarity between the stylized image output by the generator and the standard style image can be calculated. When the similarity is greater than the set similarity threshold (e.g., 90%), the training indicator is considered to be completed, thereby obtaining the target generator; when the similarity is less than the set similarity threshold, the training indicator is considered to be incomplete. At this time, additional iterative training operations can be added until the similarity meets the indicator requirements. In another method, after reaching the preset number of times, it can also be determined whether the training indicator is completed by determining whether the loss values ​​of the generator and the discriminator are within the preset range.

[0095] In summary, the present application provides a training method for an image style transfer model, and the present application can extract a first edge region of a target object in a stylized image, and a second edge region of a target object in a real image. The subsequent training process can use only the edge region as input, so that during the training process, the training process of the model can use only the edge region as input, so that the model can only process the features of the edge region, reducing the amount of calculation. In addition, the training process can focus more on the edge region fusion in the image style transfer process, so that the changes in the detailed texture of the edge region can be easily captured, thereby improving the edge region repair effect of the image stylization result.

[0096] Figure 4 is a flowchart of a method for generating a stylized image provided by an embodiment of the present application. Figure 4 As shown, the method may include:

[0097] Step 201: input the image to be processed into an image style transfer model to obtain a stylized image output by the image style transfer model.

[0098] Among them, the image style transfer model is based on Figure 1 The image style transfer model provided is obtained by the training method.

[0099] In an embodiment of the present application, after the image style transfer model is established, the image style transfer model can be embedded in the image processing process, such as a mobile terminal, a server, a cloud, etc., to provide a high-quality stylized processing function, so that the user can use the image to be processed as the input of the image style transfer model, add stylized features to the image to be processed, and obtain a stylized image with high stylization quality and fusion quality.

[0100] In summary, the present application provides a training method for an image style transfer model, and the present application can extract a first edge region of a target object in a stylized image, and a second edge region of a target object in a real image. The subsequent training process can use only the edge region as input, so that during the training process, the training process of the model can use only the edge region as input, so that the model can only process the features of the edge region, reducing the amount of calculation. In addition, the training process can focus more on the edge region fusion in the image style transfer process, so that the changes in the detailed texture of the edge region can be easily captured, thereby improving the edge region repair effect of the image stylization result.

[0101] Figure 5 is a flowchart of the steps of another method for training an image style transfer model provided in an embodiment of the present application, such as Figure 5 As shown, including:

[0102] Step 301: Obtain an original image and a real image.

[0103] The original image and the real image both contain a target object, and a style attribute of the target object contained in the original image is different from a style attribute of the target object contained in the real image;

[0104] The implementation method of this step is similar to the implementation process of the above-mentioned step 101, and the embodiment of the present application will not be described in detail here.

[0105] Step 302: Input the original image into the generator of the neural network model to obtain a stylized image, wherein the stylized image is an image obtained after stylizing the target object in the original image; the neural network model also includes a discriminator.

[0106] The implementation method of this step is similar to the implementation process of the above-mentioned step 102, and the embodiment of the present application will not be described in detail here.

[0107] Step 303: extract a first edge region of the target object in the stylized image and a second edge region of the target object in the real image.

[0108] The implementation method of this step is similar to the implementation process of the above-mentioned step 103, and the embodiment of the present application will not be described in detail here.

[0109] Optionally, step 303 may specifically include:

[0110] Sub-step 3031: determining a first boundary line at the junction of a style region and an adjacent region of the target object in the stylized image, and a second boundary line at the junction of a target region and an adjacent region of the target object in the real image.

[0111] In the embodiment of the present application, before extracting the edge region in the image, a first boundary line at the junction of the style region and the adjacent region in the stylized image and a second boundary line at the junction of the target region and the adjacent region in the real image can be determined. Figure 3 , the first dividing line in the stylized image is the boundary between the region 10 as the style region and the adjacent region 20 .

[0112] Sub-step 3032: In the stylized image, use the first dividing line as the center line of the first edge region to divide the image into a first edge region of a preset width.

[0113] In the embodiment of the present application, after the first dividing line is determined, the first dividing line can be used as the center line of the first edge area, and the preset pixel distance can be extended in two directions away from the first dividing line to divide the extended area into the first edge area.

[0114] Sub-step 3033: In the real image, use the second dividing line as the center line of the second edge area to divide the second edge area of ​​the preset width.

[0115] In the embodiment of the present application, after the second dividing line is determined, the second dividing line can be used as the center line of the second edge area, and the preset pixel distance can be extended in two directions away from the second dividing line to divide the extended area into the second edge area. The specific value of the preset pixel distance can be set according to actual needs, and the embodiment of the present application does not limit this.

[0116] Step 304: perform image segmentation on the first edge region to obtain a plurality of first image blocks of preset sizes.

[0117] Step 305: perform image segmentation on the second edge area to obtain a plurality of second image blocks of the preset size.

[0118] In the embodiment of the present application, since the first edge region and the second edge region are relatively dispersed, narrow and irregular, and the discriminator has a better processing effect on rectangular images, the first edge region and the second edge region can be further segmented into multiple first image blocks and second image blocks with m×n pixels, and each image block can constitute a rectangular image, so that the first image block and the second image block can be used as training inputs of the discriminator later, thereby meeting the purpose of the discriminator processing rectangular images. Among them, the specific values ​​of the pixel size m×n of the first image block and the second image block can be set according to actual needs, and the embodiment of the present application does not limit this.

[0119] For example, refer to Figure 6 , which shows a schematic diagram of edge area segmentation provided by an embodiment of the present application, for Figure 3 The first edge region 30 in the stylized image shown can be segmented into a plurality of rectangular first image blocks 31 through an image segmentation operation, and each first image block 31 has the same pixel size.

[0120] Furthermore, after the first edge region and the second edge region are divided into smaller image blocks, the subsequent training process of the discriminator and the generator can focus more on identifying the fusion quality in finer-grained image blocks, further improving the training accuracy of the discriminator and the generator.

[0121] Step 306: train the generator and the discriminator according to the first image block and the second image block, and use the trained generator as the image style transfer model.

[0122] In an embodiment of the present application, the parameters of the discriminator can be trained by taking the first image block included in the first edge region and the second image block included in the second edge region as input, and then taking the first image block as the input of the discriminator, and using the output value of the discriminator and the true value (the true value at this time is that the discriminator can identify the first edge region as "true") to calculate the loss value, and then using the loss value and the loss function to train the parameters of the generator. After the edge region is divided into image blocks for training, since each image block can reflect a part of the fusion details in the edge region, the capture of the fusion details of the edge region during the training process of the image style transfer model can be further improved, thereby further improving the training effect.

[0123] Optionally, in one implementation, step 306 may include:

[0124] Sub-step 3061: training the discriminator according to the first image block and the second image block.

[0125] Sub-step 3062: training the generator according to the first image block and the discriminator.

[0126] In one implementation, for an iterative operation, the discriminator can be trained with the first image block and the second image block as the training goal, and the parameters of the discriminator can be trained using the first image block and the second image block. Then, the first block is input into the discriminator with improved ability after training, and the first block is trained with the discriminator being able to identify the first block as "true" as the training goal. The parameters of the generator are trained using the output value, true value, and loss function of the discriminator with improved ability after training. After training, the fusion quality of the first edge area in the stylized image generated by the generator may be improved.

[0127] After multiple rounds of iterations, both the generator and the discriminator can get closer to the training indicators.

[0128] Optionally, in another implementation, step 306 may include:

[0129] Sub-step 3063: training the generator according to the first image block and the discriminator.

[0130] Sub-step 3064: training the discriminator according to the first image block and the second image block.

[0131] In another implementation, for one iteration operation, the first block can be input into the discriminator first, and the training purpose is that the first block can be judged as "true" by the discriminator. The output value, true value, and loss function of the discriminator are used to train the parameters of the generator. After training, the fusion quality of the first edge area in the stylized image generated by the generator may be improved. Then, the training purpose is that the discriminator can identify the first image block as "fake" and the second image block as "true", and the first image block and the second image block are used to train the parameters of the discriminator, thereby improving the discriminator's ability to distinguish the true and false edge areas.

[0132] After multiple rounds of iterations, both the generator and the discriminator can get closer to the training indicators.

[0133] Optionally, the training of the discriminator according to the first image block and the second image block may be specifically implemented by the following sub-steps:

[0134] Sub-step A1: input the first image block and the second image block into the discriminator respectively, and obtain a first determination result corresponding to the first image block and a second determination result corresponding to the second image block.

[0135] Sub-step A2: determining a first loss value according to a difference between a first true value of the first image block and the first judgment result, and determining a second loss value according to a difference between a true value of the second image block and the second judgment result; the first true value reflects the value taken when the judgment result of the discriminator on the input object is false; the true value of the second image block reflects the value taken when the judgment result of the discriminator on the input object is true.

[0136] Sub-step A3: alternately training the discriminator according to the first loss value and the second loss value.

[0137] Specifically, the result output by the discriminator is a real number in the range of 0 to 1. If the output value of the discriminator is 0.5-1, it means that the judgment is true at this time; if the output value of the discriminator is 0-0.5, it means that the judgment is false at this time.

[0138] Since during the training of the discriminator, the training goal for the first image block is that the discriminator can identify the result of the first image block as "false", and the training goal for the second image block is that the discriminator can identify the result of the second image block as "true", then the first true value corresponding to the first image block is 0, and the true value corresponding to the second image block is 1, which makes the first loss value of the first image block the difference between the first true value 0 and the first result corresponding to the first image block, and the second loss value of the second image block is the difference between the true value 1 and the second result corresponding to the second image block. The parameters of the discriminator are alternately trained using the first loss value and the second loss value. After multiple rounds of iterative operations, the ability of the discriminator to identify the true and false image blocks is improved. Among them, alternating training means training the parameters of the discriminator with the first loss value once and training the parameters of the discriminator with the second loss value another time.

[0139] The loss function is a function that maps the value of a random variable to a non-negative real number to represent the "risk" or "loss" of the random event. In applications, the loss function is usually associated with the optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function. In the process of training the model, a suitable loss function can be selected according to actual needs. The embodiment of the present application does not limit the specific first loss function selected.

[0140] During the training process, the stochastic gradient descent algorithm can also be used to optimize the training process. The stochastic gradient descent algorithm randomly extracts a set of data from the training data in multiple iterations, updates it once according to the gradient after training, and then extracts another set of data and updates it again. When the amount of training data is very large, it may not be necessary to train all the samples to obtain a model with a loss value within an acceptable range. Among them, random means that the input data will be randomly shuffled during each iteration, and its purpose is to effectively reduce the parameter update offset problem caused by the input data. After each iteration, the loss value between the output result of the discriminator and the true value will decrease by gradient. When the loss value is within the preset range, it can be considered that the training of the image style transfer model is completed.

[0141] Optionally, the training of the generator according to the first image block and the discriminator may be specifically implemented by the following sub-steps:

[0142] Sub-step B1: input the first image block into the discriminator to obtain a third determination result corresponding to the first image block.

[0143] Sub-step B2: determining a third loss value according to a difference between a second true value of the first image block and the third judgment result; the second true value reflects the value taken when the judgment result of the discriminator on the input object is true.

[0144] Sub-step B3: training the generator according to the third loss value.

[0145] In the process of training the generator, the training goal is that the discriminator can identify the result of the first image block in the stylized image output by the generator as "true", then the second true value corresponding to the first image block is 1, which makes the third loss value of the first image block the difference between the second true value 1 and the third result corresponding to the first image block. The parameters of the generator are trained using the third loss value and the second loss function. After multiple rounds of iterative operations, the generator's ability to output stylized images with high fusion quality edge areas is improved. After reaching the preset number of iterations, the training of the image style transfer model is completed, and the target generator is obtained.

[0146] In summary, the present application provides a training method for an image style transfer model, and the present application can extract a first edge region of a target object in a stylized image, and a second edge region of a target object in a real image. The subsequent training process can use only the edge region as input, so that during the training process, the training process of the model can use only the edge region as input, so that the model can only process the features of the edge region, reducing the amount of calculation. In addition, the training process can focus more on the edge region fusion in the image style transfer process, so that the changes in the detailed texture of the edge region can be easily captured, thereby improving the edge region repair effect of the image stylization result.

[0147] Figure 7 is a block diagram of a training device for an image style transfer model provided in an embodiment of the present application, such as Figure 7 As shown, it includes: an acquisition module 401, a first style module 402, a segmentation module 403, and a training module 404.

[0148] The acquisition module 401 is configured to acquire an original image and a real image, wherein both the original image and the real image contain a target object, and the style attribute of the target object contained in the original image is different from the style attribute of the target object contained in the real image;

[0149] A first style module 402 is configured to input the original image into a generator of a neural network model to obtain a stylized image, wherein the stylized image is an image obtained by stylizing a target object in the original image; the neural network model further includes a discriminator;

[0150] A segmentation module 403 is configured to extract a first edge region of the target object in the stylized image and a second edge region of the target object in the real image;

[0151] The training module 404 is configured to train the generator and the discriminator according to the first edge region and the second edge region, and use the trained generator as an image style transfer model.

[0152] The training module includes:

[0153] A first segmentation submodule is configured to perform image segmentation on the first edge area to obtain a plurality of first image blocks of preset sizes;

[0154] A second segmentation submodule is configured to perform image segmentation on the second edge area to obtain a plurality of second image blocks of the preset size;

[0155] A training submodule is configured to train the generator and the discriminator according to the first image block and the second image block, and use the trained generator as the image style transfer model.

[0156] In one achievable manner, the training submodule includes:

[0157] A first training unit is configured to train the discriminator according to the first image block and the second image block;

[0158] The second training unit is configured to train the generator according to the first image block and the discriminator.

[0159] In an implementable manner, the first training unit includes:

[0160] A first training subunit is configured to input the first image block and the second image block into the discriminator respectively, to obtain a first determination result corresponding to the first image block and a second determination result corresponding to the second image block;

[0161] a second training subunit, configured to determine a first loss value according to a difference between a first true value of the first image block and the first determination result, and to determine a second loss value according to a difference between a true value of the second image block and the second determination result; the first true value reflects a value taken when a determination result of the discriminator on the input object is false; and the true value of the second image block reflects a value taken when a determination result of the discriminator on the input object is true;

[0162] The alternating training subunit is configured to alternately train the discriminator according to the first loss value and the second loss value.

[0163] In an implementable manner, the second training unit includes:

[0164] a third training subunit, configured to input the first image block into the discriminator to obtain a third determination result corresponding to the first image block;

[0165] a fourth training subunit, configured to determine a third loss value according to a difference between a second true value of the first image block and the third determination result; the second true value reflects a value taken when the determination result of the discriminator on the input object is true;

[0166] A training subunit is configured to train the generator according to the third loss value.

[0167] In one achievable manner, the training module includes:

[0168] The iterative training submodule is configured to terminate the training after performing a preset number of iterative training operations on the generator and the discriminator, and use the trained generator as the image style transfer model. One iterative training operation includes training the generator once and training the discriminator once.

[0169] In summary, the present application provides a training device for an image style transfer model, and the present application can extract a first edge region of a target object in a stylized image, and a second edge region of a target object in a real image. The subsequent training process can use only the edge region as input, so that during the training process, the training process of the model can use only the edge region as input, so that the model can only process the features of the edge region, reducing the amount of calculation. In addition, the training process can focus more on the edge region fusion in the image style transfer process, so that the changes in the detailed texture of the edge region can be easily captured, thereby improving the edge region repair effect of the image stylization result.

[0170] Figure 8 is a block diagram of a stylized image generation device provided in an embodiment of the present application, such as Figure 8 As shown, it includes: a style processing module 501.

[0171] The style processing module 501 is configured to input the image to be processed into the image style transfer model to obtain a stylized image output by the image style transfer model;

[0172] Among them, the image style transfer model is based on Figure 7 The image style transfer model is shown in the training setup.

[0173] In summary, the embodiment of the present application provides a stylized image generation device, which can extract the first edge area of ​​the target object in the stylized image and the second edge area of ​​the target object in the real image. The subsequent training process can only use the edge area as input, so that during the training process, the training process of the model can only use the edge area as input, so that the model can only process the features of the edge area, reducing the amount of calculation. In addition, the training process can focus more on the edge area fusion in the image style transfer process, so that the changes in the detailed texture of the edge area can be easily captured, thereby improving the edge area repair effect of the image stylization result.

[0174] Fig. 9 is a block diagram of an electronic device 600 according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0175] Reference Fig. 9, the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0176] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0177] The memory 604 is used to store various types of data to support the operation of the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0178] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0179] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0180] The audio component 610 is used to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0181] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0182] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of the components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of contact between the user and the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0183] The communication component 616 is used to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0184] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement a training method for an image style transfer model provided in an embodiment of the present application.

[0185] In an exemplary embodiment, a non-transitory computer storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of an electronic device 600 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0186] Fig.10 is a block diagram of an electronic device 700 according to an exemplary embodiment. For example, the electronic device 700 may be provided as a server. Fig.10 The electronic device 700 includes a processing component 722, which further includes one or more processors, and a memory resource represented by a memory 732 for storing instructions executable by the processing component 722, such as an application. The application stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 722 is configured to execute instructions to perform a training method for an image style transfer model provided in an embodiment of the present application.

[0187] The electronic device 700 may also include a power supply component 726 configured to perform power management of the electronic device 700, a wired or wireless network interface 750 configured to connect the electronic device 700 to a network, and an input / output (I / O) interface 758. The electronic device 700 may operate based on an operating system stored in the memory 732, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0188] An embodiment of the present application also provides a computer program product, including a computer program, which implements the training method of the image style transfer model when executed by a processor.

[0189] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0190] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A training method for an image style transfer model, characterized in that: The method comprises: Acquire an original image and a real image, wherein both the original image and the real image contain a target object, and a style attribute of the target object contained in the original image is different from a style attribute of the target object contained in the real image; Inputting the original image into a generator of a neural network model to obtain a stylized image, wherein the stylized image is an image obtained by stylizing a target object in the original image; the neural network model further includes a discriminator; Extracting a first edge region of the target object in the stylized image and a second edge region of the target object in the real image; According to the first edge region and the second edge region, training the generator and the discriminator, and using the trained generator as an image style transfer model; The step of training the generator and the discriminator according to the first edge region and the second edge region, and using the trained generator as an image style transfer model, comprises: Performing image segmentation on the first edge area to obtain a plurality of first image blocks of preset sizes; Performing image segmentation on the second edge area to obtain a plurality of second image blocks of the preset size; Training the generator and the discriminator according to the first image block and the second image block, and using the trained generator as the image style transfer model; The step of training the generator and the discriminator according to the first image block and the second image block comprises: Training the discriminator according to the first image block and the second image block; The generator is trained according to the first image block and the discriminator.

2. The method according to claim 1, characterized in that The step of training the discriminator according to the first image block and the second image block comprises: Inputting the first image block and the second image block into the discriminator respectively, obtaining a first determination result corresponding to the first image block and a second determination result corresponding to the second image block; Determine a first loss value according to a difference between a first true value of the first image block and the first determination result, and determine a second loss value according to a difference between a true value of the second image block and the second determination result; the first true value reflects a value taken when a determination result of the discriminator on the input object is false; the true value of the second image block reflects a value taken when a determination result of the discriminator on the input object is true; The discriminator is trained alternately according to the first loss value and the second loss value.

3. The method according to claim 1, characterized in that The step of training the generator according to the first image block and the discriminator comprises: Inputting the first image block into the discriminator to obtain a third determination result corresponding to the first image block; Determine a third loss value according to a difference between a second true value of the first image block and the third determination result; the second true value reflects a value when the determination result of the discriminator on the input object is true; The generator is trained according to the third loss value.

4. The method according to claim 1, characterized in that: The step of training the generator and the discriminator according to the first edge region and the second edge region, and using the trained generator as an image style transfer model, comprises: After performing a preset number of iterative training operations on the generator and the discriminator, the training is terminated, and the trained generator is used as an image style transfer model. One iterative training operation includes training the generator once and training the discriminator once.

5. A method for generating a stylized image, characterized in that: The method comprises: Inputting the image to be processed into an image style transfer model to obtain a stylized image output by the image style transfer model; The image style transfer model is obtained by training according to the training method of the image style transfer model in any one of claims 1 to 2.

6. A training device for an image style transfer model, characterized in that: The device comprises: an acquisition module, configured to acquire an original image and a real image, wherein both the original image and the real image contain a target object, and a style attribute of the target object contained in the original image is different from a style attribute of the target object contained in the real image; A first style module is configured to input the original image into a generator of a neural network model to obtain a stylized image, wherein the stylized image is an image obtained by stylizing a target object in the original image; the neural network model further includes a discriminator; a segmentation module, configured to extract a first edge region of the target object in the stylized image and a second edge region of the target object in the real image; A training module, configured to train the generator and the discriminator according to the first edge region and the second edge region, and use the trained generator as an image style transfer model; The training module includes: A first segmentation submodule is configured to perform image segmentation on the first edge area to obtain a plurality of first image blocks of preset sizes; A second segmentation submodule is configured to perform image segmentation on the second edge area to obtain a plurality of second image blocks of the preset size; A training submodule, configured to train the generator and the discriminator according to the first image block and the second image block, and use the trained generator as the image style transfer model; The training submodule comprises: A first training unit is configured to train the discriminator according to the first image block and the second image block; The second training unit is configured to train the generator according to the first image block and the discriminator.

7. The device according to claim 6, characterized in that The first training unit comprises: A first training subunit is configured to input the first image block and the second image block into the discriminator respectively, to obtain a first determination result corresponding to the first image block and a second determination result corresponding to the second image block; a second training subunit, configured to determine a first loss value according to a difference between a first true value of the first image block and the first determination result, and to determine a second loss value according to a difference between a true value of the second image block and the second determination result; the first true value reflects a value taken when a determination result of the discriminator on the input object is false; and the true value of the second image block reflects a value taken when a determination result of the discriminator on the input object is true; The alternating training subunit is configured to alternately train the discriminator according to the first loss value and the second loss value.

8. The device according to claim 7, characterized in that The second training unit comprises: a third training subunit, configured to input the first image block into the discriminator to obtain a third determination result corresponding to the first image block; a fourth training subunit, configured to determine a third loss value according to a difference between a second true value of the first image block and the third determination result; the second true value reflects a value taken when the determination result of the discriminator on the input object is true; A training subunit is configured to train the generator according to the third loss value.

9. The device according to claim 6, characterized in that The training module comprises: The iterative training submodule is configured to terminate the training after performing a preset number of iterative training operations on the generator and the discriminator, and use the trained generator as the image style transfer model. One iterative training operation includes training the generator once and training the discriminator once.

10. A stylized image generation device, characterized in that: The device comprises: A style processing module, configured to input the image to be processed into an image style transfer model to obtain a stylized image output by the image style transfer model; Wherein, the image style transfer model is obtained according to the training device of the image style transfer model according to claim 6.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method as claimed in any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 5.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images