Training method of image style migration model, style migration method and related device

Abstract

One or more embodiments of the invention disclose an image style migration model training method, a style migration method and a related device, and the method comprises the steps: carrying out the segmentation and zooming processing of a content image, and obtaining a plurality of content image blocks and a thumbnail content image, and inputting the content image block, the thumbnail content image and the style image into a preset network model, determining perception loss based on the generated stylized content image block and stylized thumbnail content image, and optimizing and adjusting the preset network model based on the perception loss and adversarial loss to obtain an image style migration model. As the content image is subjected to segmentation processing, the method can be suitable for any high-resolution image migration processing, and the universality of a style migration scheme is improved. Moreover, the perception loss is determined by combining the thumbnail content image, so that the image style migration model can be guided to generate a style migration image with thicker lines and sparse textures, and the high-resolution image effect is improved.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a training method for an image style transfer model, a style transfer method, and related devices. Background Art

[0002] As people continue to delve deeper into the field of interior design, various advanced home decoration design software has emerged. In the current interior design process, renderings or real-life images are usually used directly as materials. In particular, when facing various design requirements, some images of other styles may be needed to enrich the materials.

[0003] In the prior art, a large number of images with a target style can be generated through style transfer. However, in home style design scenarios with complex scene contents, most style transfer schemes cannot achieve the ideal transfer effect, and the generated style transfer images have poor image quality, and even distortion and artifacts occur. Summary of the invention

[0004] The purpose of one or more embodiments of the present specification is to provide a training method for an image style transfer model, a style transfer method, and related devices to avoid distortion and artifacts in images after style transfer, and to improve style transfer efficiency and image quality.

[0005] To solve the above technical problems, one or more embodiments of this specification are implemented as follows:

[0006] First, a training method for an image style transfer model is proposed, including:

[0007] Obtain content images and style images from the sample set;

[0008] The content image is segmented according to a preset segmentation rule to obtain a plurality of content image blocks with the same resolution; and the content image is scaled to obtain a thumbnail content image, wherein the resolution of the thumbnail content image is the same as the resolution of the content image block;

[0009] Inputting the content image block, the thumbnail content image, and the style image into a preset network model to obtain a stylized thumbnail content image and a stylized content image block;

[0010] Calculating a perceptual loss based on the stylized thumbnail content image and the stylized content image patch, and calculating an adversarial loss based on the stylized content image patch and the style image;

[0011] The preset network model is optimized and adjusted based on the perceptual loss and the adversarial loss to obtain an image style transfer model.

[0012] Secondly, a style transfer method is proposed, including:

[0013] Determining an original content image to be processed;

[0014] The original content image is segmented according to a preset segmentation rule to obtain a plurality of content image blocks with the same resolution;

[0015] Inputting a plurality of content image blocks into an image style transfer model to generate a plurality of stylized content image blocks, wherein a normalization parameter in the image style transfer model is determined based on a thumbnail content image of the original content image;

[0016] The multiple stylized content image blocks are spliced ​​and combined to obtain a style-transferred image; wherein the image style transfer model is obtained by the training method of the image style transfer model described in the first aspect.

[0017] Thirdly, a training device for an image style transfer model is proposed, including:

[0018] An acquisition module is used to obtain content images and style images from a sample set;

[0019] a processing module, configured to segment the content image according to a preset segmentation rule to obtain a plurality of content image blocks of the same resolution; and to scale the content image to obtain a thumbnail content image, wherein the resolution of the thumbnail content image is the same as the resolution of the content image block;

[0020] An input module, configured to input the content image block, the thumbnail content image, and the style image into a preset network model to obtain a stylized thumbnail content image and a stylized content image block;

[0021] A calculation module, configured to calculate a perceptual loss based on the stylized thumbnail content image and the stylized content image block, and calculate an adversarial loss based on the stylized content image block and the style image;

[0022] A training module is used to optimize and adjust the preset network model based on the perceptual loss and the adversarial loss to obtain an image style transfer model.

[0023] In a fourth aspect, a style transfer device is proposed, comprising:

[0024] A determination module, used for determining an original content image to be processed;

[0025] A processing module, configured to segment the original content image according to a preset segmentation rule to obtain a plurality of content image blocks with the same resolution;

[0026] A generating module, configured to input a plurality of content image blocks into an image style transfer model to generate a plurality of stylized content image blocks, wherein a normalization parameter in the image style transfer model is determined based on a thumbnail content image of the original content image;

[0027] A combination module is used to splice and combine the multiple stylized content image blocks to obtain a style transfer image; wherein the image style transfer model is obtained by the training method of the image style transfer model described in the first aspect.

[0028] In a fifth aspect, an electronic device is provided, comprising:

[0029] Processor; and

[0030] A memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the training method of the image style transfer model described in the first aspect or the style transfer method described in the second aspect.

[0031] In the sixth aspect, a computer-readable storage medium is proposed, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device executes the training method of the image style transfer model described in the first aspect or the style transfer method described in the second aspect.

[0032] It can be seen from the technical solution provided by one or more embodiments of the above specification that by segmenting and scaling the content image respectively, a plurality of content image blocks and a thumbnail content image are obtained, and the plurality of content image blocks, a thumbnail content image and a style image are input into a preset network model. On the one hand, the perceptual loss is determined based on the generated stylized content image blocks and the stylized thumbnail content image, and on the other hand, the adversarial loss is determined based on the generated stylized content image blocks and the style image. Then, the preset network model is optimized and adjusted using the determined perceptual loss and adversarial loss, so as to obtain the required image style transfer model through repeated training. Since the content image is segmented, it can be applied to any high-resolution image transfer processing, thereby improving the universality of the style transfer solution. Moreover, since the perceptual loss is determined in combination with the thumbnail content image, the image style transfer model can be guided to generate a style transfer image with thicker lines and sparser textures, thereby improving the high-resolution image effect, ignoring the image details, and paying more attention to the overall image effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the description of one or more embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 It is a schematic diagram of the steps of a training method for an image style transfer model provided in an embodiment of this specification.

[0035] Figure 2 It is a schematic diagram of the principle of segmentation and scaling of content images provided in an embodiment of this specification.

[0036] Figure 3 It is a schematic diagram of the steps of an image style transfer method provided in an embodiment of this specification.

[0037] Figure 4a and Figure 4b They are respectively schematic diagrams of an image style transfer processing flow provided by an embodiment of the present specification.

[0038] Figure 5 It is a structural diagram of a training device for an image style transfer model provided in an embodiment of this specification.

[0039] Figure 6 It is a schematic diagram of the structure of the image style transfer device provided in the embodiments of this specification.

[0040] Figure 7a and 7b It is a schematic diagram of the structure of an electronic device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the one or more embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0042] Style transfer refers to the conversion of images in two different domains. Specifically, it provides a style image and converts any content image into the same style as the style image, while retaining the content of the content image as much as possible.

[0043] Interior design is usually presented in the form of rendered or real-life pictures. In many cases, some cartoon-style pictures are needed as materials. For example, cartoon-styled renderings in children's rooms can bring an immersive experience to users; indoor home decoration growth animations combined with cartoon styles can better express the designer's design creativity. Usually, these materials need to be obtained by hand-drawing, which has the problems of high cost and low efficiency. At present, a large number of images with target styles can be generated by style transfer. However, considering that the content and style of some scenes are relatively complex, most style transfer schemes cannot achieve the ideal transfer effect. As a result, deep learning networks have begun to be used in the field of style transfer, although they are more efficient than previous methods. However, there are also some problems: First, most current style transfer systems are limited by large memory usage when processing ultra-high-resolution images (above 8K), and existing style transfer methods are often difficult to truly apply. Second, in modern home style scenes with complex scene content, the ideal effect cannot be achieved, and there are problems such as poor quality of the transfer effect, loss of beauty, distortion and artifacts.

[0044] In order to solve the above problems, this specification proposes a training method for an image style transfer model and a corresponding image style transfer method. The main inventive concept is: by segmenting and scaling the content image respectively, a plurality of content image blocks and a thumbnail content image are obtained, and the plurality of content image blocks, a thumbnail content image and a style image are input into a preset network model. On the one hand, the perceptual loss is determined based on the generated stylized content image blocks and the stylized thumbnail content image, and on the other hand, the adversarial loss is determined based on the generated stylized content image blocks and the style image. Then, the preset network model is optimized and adjusted using the determined perceptual loss and adversarial loss, so as to repeatedly train and obtain the required image style transfer model. Since the content image is segmented, it can be applied to any high-resolution image transfer processing, thereby improving the universality of the style transfer scheme. Moreover, since the perceptual loss is determined in combination with the thumbnail content image, the image style transfer model can be guided to generate a style transfer image with thicker lines and sparser textures, thereby improving the high-resolution image effect, ignoring the image details, and paying more attention to the overall image effect.

[0045] Embodiment 1

[0046] Reference Figure 1As shown, it is a schematic diagram of the steps of a training method for an image style transfer model provided in an embodiment of this specification. It should be understood that this method can be applied to various image processing scenarios involving image style transfer, such as the field of home design or the field of image processing, etc., and its execution subject can be a training device for the image style transfer model, and the training device can be a software module or hardware device with the following functions, such as a computer, a mobile phone, an application, etc., which is not limited here. The training method may include the following steps:

[0047] Step 102: Obtain content images and style images from a sample set.

[0048] In the embodiment of the present specification, the sample set may be an image set including multiple content images and style images. Considering the type restrictions of style images, the number of content images in the sample set may be much larger than the number of style images. For example, the number of content images is 5,000 and the number of style images is 2,000.

[0049] The content image here refers to the image to be stylized, and the image content of the image does not change before and after the style transfer process. The types and forms of content images are various, and this description does not limit this. For the sake of convenience, the content image can be various types of decoration images or home decoration design images involved in the field of home decoration. The style image refers to the style image of the style to be transferred when training the style transfer model. The style image and the content image here are used as training samples together and input into the style transfer model to be trained. The style of the style image can be modern style, Roman style, cartoon style or court style, etc.

[0050] Step 104: Slice the content image according to a preset segmentation rule to obtain a plurality of content image blocks with the same resolution; and scale the content image to obtain a thumbnail content image, wherein the resolution of the thumbnail content image is the same as the resolution of the content image block.

[0051] In the embodiments of the present specification, different processing can be performed on the content image: segmentation and scaling. The segmentation processing and scaling processing can be performed simultaneously, or the content image is first segmented and then the same content image is scaled; or the content image is first scaled and then the same content image is segmented.

[0052] In specific implementation, the content image can be cut according to the preset segmentation rules to obtain multiple content image blocks of the content image, and the resolutions of these content image blocks are the same; at the same time, the content image is scaled to reduce the content image to a certain resolution to obtain a thumbnail content image. It should be understood that the resolution of the content image block is the same as the resolution of the thumbnail content image, so that it can be compared in subsequent model training and some available parameters can be referenced to each other. In addition, for high-resolution content images, the image resolution of the subsequent input training model can be reduced by segmentation in advance, thereby reducing the video memory usage. Among them, the preset segmentation rules involved can be segmentation into 4 equal parts, or segmentation into 16 equal parts, etc.

[0053] Reference Figure 2 As shown, when the content image is segmented and scaled, it can be processed by the image processing module to reduce the input resolution of the subsequent style transfer module and reduce the video memory usage. The input of the image processing module is the high-resolution content image to be processed (here high resolution is used as an example, it can be low resolution or other resolutions), and the output has two parts: the segmented low-resolution content image block and the scaled low-resolution thumbnail content image. For example: a 2000x2000 resolution content image can be segmented into 4 1000x1000 resolution images by the sliding window method, and a 1000x1000 resolution thumbnail is obtained by scaling.

[0054] In addition, when scaling the content image, the image may be reduced based on equal-interval sampling or based on a local mean value, etc., and this specification does not limit this.

[0055] Step 106: Input the content image block, the thumbnail content image, and the style image into a preset network model to obtain a stylized thumbnail content image and a stylized content image block.

[0056] The preset network model may have two network parts, one part is a generator for generating a stylized image, and the other part is a discriminator for comparing and analyzing the generated stylized image with the input style image. After the content image block and the thumbnail content image are input into the preset network model, the stylized content image block and the stylized thumbnail content image are generated respectively by the generator.

[0057] Step 108: Calculate a perceptual loss based on the stylized thumbnail content image and the stylized content image block, and calculate an adversarial loss based on the stylized content image block and the style image.

[0058] After the style image is input into the preset network model, the discriminator compares the style image with the stylized content image blocks generated by the generator, mainly comparing and analyzing the authenticity and stylization degree to obtain the adversarial loss. At the same time, the perceptual loss is calculated based on the stylized thumbnail content image and stylized content image blocks generated by the generator.

[0059] In a specific implementation, the perceptual loss corresponding to the image block can be calculated based on each stylized content image block and the stylized thumbnail content image respectively; the perceptual loss corresponding to the calculated image block is weighted averaged to obtain the perceptual loss corresponding to the content image. In other words, when calculating the perceptual loss of the content image, the perceptual loss of each content image block can be calculated according to the number of segments of the content image, and then the perceptual losses of all content image blocks are weighted averaged to obtain the perceptual loss of the content image.

[0060] Further, calculating the perceptual loss corresponding to the image block based on each stylized content image block and the stylized thumbnail content image may specifically include the following steps:

[0061] First, a stylized content image block is determined; then, an area with the same content as the stylized content image block is selected from the stylized thumbnail content image, and the selected area is enlarged, wherein the resolution of the enlarged area is the same as the resolution of the stylized content image block; finally, based on the stylized content image block and the enlarged area in the stylized thumbnail content image, the perceptual loss corresponding to the stylized content image block is calculated. When enlarging the selected area of ​​the stylized thumbnail content image, the nearest neighbor interpolation method, or the bilinear interpolation method or the cubic convolution method can be used. The operation method of the image enlargement processing is not limited here, and it can be flexibly selected according to the needs.

[0062] Wherein, when calculating the perceptual loss corresponding to the stylized content image block based on the stylized content image block and the enlarged area in the stylized thumbnail content image, a first image feature can be specifically extracted from the stylized content image block, and a second image feature can be extracted from the enlarged area in the stylized thumbnail content image; and the difference between the first image feature and the second image feature is calculated to obtain the perceptual loss corresponding to the stylized content image block.

[0063] When calculating the perceptual loss, you can refer to the following formula:

[0064]

[0065] Among them, L per is the perceived loss, Refers to the training set of samples used in this training; VGG(G(t i )) refers to the image features of the i-th region corresponding to the stylized thumbnail extracted using the pre-trained VGG-19 network; VGG(G(p i )) refers to the image features of the i-th image patch extracted using the pre-trained VGG-19 network. G(pi) is a stylized patch with small brushstrokes, and G(ti) is a stylized patch with large brushstrokes, where ti is cropped and upsampled from the thumbnail. Guided by the newly proposed perceptual loss, the style generation network learns to depict the scene using thicker lines and sparser textures, which helps improve the visual effect of ultra-high resolution style transfer. When an image is reduced, some pixels are removed, reducing the resolution of the image. When the image is enlarged again, the missing pixels need to be interpolated to fill the gaps, making the strokes thicker and the details less.

[0066] In terms of adversarial loss, the adversarial loss can be calculated based on the stylized content image block and the style image. Specifically, each stylized content image block can be compared with the style image at the style perspective to calculate the adversarial loss corresponding to the image block; the adversarial loss corresponding to the calculated image block is weighted and averaged to obtain the adversarial loss corresponding to the content image.

[0067] In fact, in the process of calculating the loss function of the model network to adjust the model parameters, not only adversarial loss and perceptual loss are involved, but also style loss, color loss, content loss, total variation loss, etc. Among them, adversarial loss acts on the construction process of the cartoon style generator and is the basic loss function of the generative adversarial network; the role of perceptual loss is to improve the quality of ultra-high resolution style transfer. The core of this loss is: to penalize the perceptual difference between the stroke edge of the cartoonized image block and the upsampled cartoonized thumbnail; the role of style loss is to make the style of the stylized image similar to that of the cartoon image. It should be noted that the cartoon image is converted from RGB channels to YUV channels, where the Y channel represents brightness, and the U and V channels represent color information. Here, only the Y channel is calculated for style loss to retain the texture and lines of the cartoon image. The role of color loss is to make the color of the stylized image similar to that of the input image, which is achieved by comparing the U and V channels of the stylized image and the input image; the role of content loss is to make the content of the stylized image similar to that of the input image, and maintain the consistency of high-dimensional semantic information; the role of total variation loss is to make the image smoother and solve the problem of high-frequency artifacts in the generated image. Considering that this manual only improves the two dimensions of adversarial loss and perceptual loss, other loss functions are not introduced in detail here.

[0068] Step 110: Optimize and adjust the preset network model based on the perceptual loss and the adversarial loss to obtain an image style transfer model.

[0069] After determining the perceptual loss and adversarial loss through the above scheme, the preset network model can be optimized and adjusted based on the perceptual loss and adversarial loss and other loss functions to obtain the required image style transfer model. Because the feature information of the thumbnail prompts the network to generate thicker lines and sparser textures, the high-resolution cartoon effect is optimized.

[0070] Reference Figure 3 As shown in the figure, it is a schematic diagram of the steps of an image style transfer method provided in an embodiment of the present specification. It should be understood that the method can be applied to various image processing scenarios involving image style transfer, such as the field of home design or the field of image processing, etc., and its execution subject can be an image style transfer device, which can be equipped with the following Figure 3 The software module or hardware device of the function shown, for example, a computer, a mobile phone, an application, etc., is not limited here. The image style transfer method may include the following steps:

[0071] Step 302: Determine the original content image to be processed.

[0072] The original content image here may be a content image to be stylized, and its specific content and form are not limited.

[0073] Step 304: The original content image is segmented according to a preset segmentation rule to obtain a plurality of content image blocks with the same resolution.

[0074] For the original content image Figure 2 The segmentation process is performed in the manner shown, with no limit on the specific segmentation rules and the number of segmented image blocks, to obtain multiple content image blocks with the same resolution.

[0075] Optionally, after the original content image is cut in step 304, the original content image may be scaled to obtain a thumbnail content image, wherein the resolution of the thumbnail content image is the same as the resolution of the content image block. In this way, before inputting multiple content image blocks into the image style transfer model to generate multiple stylized content image blocks, the thumbnail content image may be input into the image style transfer model to determine the normalization parameters of the image style transfer model when the thumbnail content image is used as input.

[0076] Step 306: Input the plurality of content image blocks into an image style transfer model to generate a plurality of stylized content image blocks, wherein a normalization parameter in the image style transfer model is determined based on a thumbnail content image of the original content image.

[0077] Optionally, multiple content image blocks are input into an image style transfer model to generate multiple stylized content image blocks. Specifically, multiple content image blocks can be input into an image style transfer model based on the normalization parameter setting to generate multiple stylized content image blocks. The normalization parameter is calculated by the following formula:

[0078] Among them, IN(x, t) is the normalization parameter; γ is the scaling variable, β is the translation variable, x is the parameter in the model, is the normalization parameter, where μ(t) is the mean of each image calculated along the channel; σ(t) is the variance of each image calculated along the channel.

[0079] Step 308: The plurality of stylized content image blocks are combined to obtain a style-transferred image; wherein the image style transfer model is implemented by Figure 1 The training method of the image style transfer model shown is obtained.

[0080] Considering that image stitching includes image matching and image fusion, when performing image stitching and combination processing, image matching can be performed based on the spatial domain or frequency domain, and image fusion can be performed based on smooth transition or optimal seams. The specific selection can be flexibly based on actual service needs and is not limited here.

[0081] This solution can include two parts: the training phase of the image style transfer model and the image style transfer phase; refer to Figure 4a As shown, the content image and style image as samples are input into the training module for image processing training. For details, refer to the above method steps. After repeated training, the image style transfer model is obtained. Figure 4b As shown in the figure, in the image style transfer stage, the original image is cut and scaled, and then input into the image style transfer model for stylization processing, and the stylized image blocks are output, and then spliced ​​into a complete stylized image. It can be seen that this solution can perform style rendering processing on a series of stylized content images such as interior design, and can stably and efficiently process images of any resolution, avoiding problems such as insufficient hardware video memory and incorrect content transfer.

[0082] It should be noted that the user information (including but not limited to user device information, user personal information, user apartment information, user image information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0083] Embodiment 2

[0084] Reference Figure 5 As shown, a training device for an image style transfer model provided in an embodiment of this specification may include:

[0085] An acquisition module 502 is used to acquire a content image and a style image from a sample set;

[0086] The processing module 504 is used to cut the content image according to a preset cutting rule to obtain a plurality of content image blocks with the same resolution; and to scale the content image to obtain a thumbnail content image, wherein the resolution of the thumbnail content image is the same as the resolution of the content image block;

[0087] An input module 506, configured to input the content image block, the thumbnail content image, and the style image into a preset network model to obtain a stylized thumbnail content image and a stylized content image block;

[0088] A calculation module 508, configured to calculate a perceptual loss based on the stylized thumbnail content image and the stylized content image block, and to calculate an adversarial loss based on the stylized content image block and the style image;

[0089] The training module 510 is used to optimize and adjust the preset network model based on the perceptual loss and the adversarial loss to obtain an image style transfer model.

[0090] Optionally, as an embodiment, when calculating the perceptual loss based on the stylized thumbnail content image and the stylized content image block, the calculation module 508 is specifically used to: calculate the perceptual loss corresponding to the image block based on each stylized content image block and the stylized thumbnail content image respectively; and perform weighted averaging on the perceptual losses corresponding to the calculated image blocks to obtain the perceptual loss corresponding to the content image.

[0091] In a specific implementation of the embodiments of the present specification, when the calculation module 508 calculates the perceptual loss corresponding to the image block based on each stylized content image block and the stylized thumbnail content image respectively, it is specifically used to: determine the stylized content image block; select an area with the same content as the stylized content image block from the stylized thumbnail content image, and enlarge the selected area, wherein the resolution of the enlarged area is the same as the resolution of the stylized content image block; and calculate the perceptual loss corresponding to the stylized content image block based on the stylized content image block and the enlarged area in the stylized thumbnail content image.

[0092] In another specific implementation of the embodiments of the present specification, when calculating the perceptual loss corresponding to the stylized content image block based on the stylized content image block and the enlarged area in the stylized thumbnail content image, the calculation module 508 is specifically used to: extract a first image feature from the stylized content image block, and extract a second image feature from the enlarged area in the stylized thumbnail content image; calculate the difference between the first image feature and the second image feature to obtain the perceptual loss corresponding to the stylized content image block.

[0093] In another specific implementation of the embodiments of the present specification, when the calculation module 508 calculates the adversarial loss based on the stylized content image block and the style image, it is specifically used to: compare each stylized content image block with the style image at the style perspective, and calculate the adversarial loss corresponding to the image block; and perform weighted averaging on the adversarial losses corresponding to the calculated image blocks to obtain the adversarial loss corresponding to the content image.

[0094] Reference Figure 6 As shown, a style transfer device provided in an embodiment of this specification may include:

[0095] A determination module 602, used to determine an original content image to be processed;

[0096] A processing module 604 is used to segment the original content image according to a preset segmentation rule to obtain a plurality of content image blocks with the same resolution;

[0097] A generating module 606, configured to input a plurality of content image blocks into an image style transfer model to generate a plurality of stylized content image blocks, wherein a normalization parameter in the image style transfer model is determined based on a thumbnail content image of the original content image;

[0098] The combination module 608 is used to splice and combine the multiple stylized content image blocks to obtain a style transfer image; wherein the image style transfer model is obtained by the image style transfer model training method described in steps 102 to 110.

[0099] Optionally, as an embodiment, the processing module 604 is further used to: perform scaling processing on the original content image to obtain a thumbnail content image, wherein the resolution of the thumbnail content image is the same as the resolution of the content image block;

[0100] The style transfer apparatus further includes: a determination module, which is used to input the thumbnail content image into the image style transfer model before the generation module 606 inputs the multiple content image blocks into the image style transfer model to generate multiple stylized content image blocks, and determine the normalization parameters of the image style transfer model when the thumbnail content image is used as input.

[0101] In a specific implementation of the embodiments of the present specification, when the generation module 606 inputs multiple content image blocks into the image style transfer model to generate multiple stylized content image blocks, it is specifically used to: input multiple content image blocks into the image style transfer model based on the normalization parameter setting to generate multiple stylized content image blocks.

[0102] By segmenting and scaling the content image, multiple content image blocks and a thumbnail content image are obtained. The multiple content image blocks, a thumbnail content image and a style image are input into a preset network model. On the one hand, the perceptual loss is determined based on the generated stylized content image blocks and stylized thumbnail content images. On the other hand, the adversarial loss is determined based on the generated stylized content image blocks and style images. Then, the preset network model is optimized and adjusted using the determined perceptual loss and adversarial loss, so as to obtain the required image style transfer model through repeated training. Since the content image is segmented, it can be applied to any high-resolution image transfer processing, which improves the universality of the style transfer scheme. Moreover, since the perceptual loss is determined in combination with the thumbnail content image, the image style transfer model can be guided to generate a style transfer image with thicker lines and sparser textures, which improves the high-resolution image effect, ignores the image details, and pays more attention to the overall image effect.

[0103] Embodiment 3

[0104] Figure 7a and Figure 7b 7 is a schematic diagram of the structure of an electronic device of an embodiment of the present specification. Please refer to FIG7. At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device may also include hardware required for other services.

[0105] The processor, the network interface and the memory may be interconnected via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, FIG7 only uses one bidirectional arrow, but does not mean that there is only one bus or one type of bus.

[0106] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0107] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a training device for the image style transfer model at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0108] A content image and a style image are obtained from a sample set; the content image is segmented according to a preset segmentation rule to obtain a plurality of content image blocks with the same resolution; and the content image is scaled to obtain a thumbnail content image, wherein the resolution of the thumbnail content image is the same as the resolution of the content image block; the content image block, the thumbnail content image, and the style image are input into a preset network model to obtain a stylized thumbnail content image and a stylized content image block; a perceptual loss is calculated based on the stylized thumbnail content image and the stylized content image block, and an adversarial loss is calculated based on the stylized content image block and the style image; and the preset network model is optimized and adjusted based on the perceptual loss and the adversarial loss to obtain an image style transfer model.

[0109] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a style transfer device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0110] Determine an original content image to be processed; cut the original content image according to a preset segmentation rule to obtain a plurality of content image blocks with the same resolution; input the plurality of content image blocks into an image style transfer model to generate a plurality of stylized content image blocks, wherein a normalization parameter in the image style transfer model is determined based on a thumbnail content image of the original content image; splice and combine the plurality of stylized content image blocks to obtain a style transferred image; wherein the image style transfer model is obtained by the above-mentioned image style transfer model training method.

[0111] The above is as in this manual Figure 1 The method performed by the device disclosed in the embodiment shown in or 3 can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in software form. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in one or more embodiments of this specification can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in one or more embodiments of this specification can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0112] The electronic device may also perform Figure 1 or 3, and implement the corresponding device in Figure 1 The functions of the embodiments shown in or 3 are not described in detail in this specification.

[0113] Of course, in addition to software implementation, the electronic device of the embodiments of this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0114] Embodiment 4

[0115] The embodiment of the present specification also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 1 Or the method of the embodiment shown in 3, and is specifically used to perform the following method:

[0116] Acquire content images and style images from a sample set; cut the content image according to a preset segmentation rule to obtain multiple content image blocks of the same resolution; and scale the content image to obtain a thumbnail content image, wherein the resolution of the thumbnail content image is the same as the resolution of the content image block; input the content image block, the thumbnail content image, and the style image into a preset network model to obtain a stylized thumbnail content image and a stylized content image block; calculate the perceptual loss based on the stylized thumbnail content image and the stylized content image block, and calculate the adversarial loss based on the stylized content image block and the style image; optimize and adjust the preset network model based on the perceptual loss and the adversarial loss to obtain an image style transfer model. Or,

[0117] Determine an original content image to be processed; cut the original content image according to a preset segmentation rule to obtain a plurality of content image blocks with the same resolution; input the plurality of content image blocks into an image style transfer model to generate a plurality of stylized content image blocks, wherein a normalization parameter in the image style transfer model is determined based on a thumbnail content image of the original content image; splice and combine the plurality of stylized content image blocks to obtain a style transferred image; wherein the image style transfer model is obtained by the above-mentioned image style transfer model training method.

[0118] In short, the above description is only a preferred embodiment of this specification and is not intended to limit the protection scope of this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the protection scope of this specification.

[0119] The systems, devices, modules or units described in one or more of the above embodiments may be implemented by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0120] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0121] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0122] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0123] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A training method for an image style transfer model, comprising: Obtain content images and style images from the sample set; Cutting the content image according to a preset cutting rule to obtain a plurality of content image blocks with the same resolution; and scaling the content image to obtain a thumbnail content image, wherein the resolution of the thumbnail content image is the same as the resolution of the content image block; Inputting the content image block, the thumbnail content image, and the style image into a preset network model to obtain a stylized thumbnail content image and a stylized content image block; Calculating a perceptual loss based on the stylized thumbnail content image and the stylized content image patch, and calculating an adversarial loss based on the stylized content image patch and the style image; The preset network model is optimized and adjusted based on the perceptual loss and the adversarial loss to obtain an image style transfer model.

2. The method for training an image style transfer model according to claim 1, wherein the perceptual loss is calculated based on the stylized thumbnail content image and the stylized content image block, specifically comprising: Calculating the perceptual loss corresponding to the image block based on each stylized content image block and the stylized thumbnail content image respectively; The calculated perceptual losses corresponding to the image blocks are weighted and averaged to obtain the perceptual loss corresponding to the content image.

3. The training method of the image style transfer model according to claim 2, wherein the perceptual loss corresponding to each stylized content image block and the stylized thumbnail content image is calculated based on each stylized content image block and the stylized thumbnail content image, specifically comprising: determining a stylized content image block; Selecting an area having the same content as the stylized content image block from the stylized thumbnail content image, and enlarging the selected area, wherein the resolution of the enlarged area is the same as the resolution of the stylized content image block; Based on the stylized content image block and the enlarged region in the stylized thumbnail content image, a perceptual loss corresponding to the stylized content image block is calculated.

4. The training method of the image style transfer model according to claim 3, wherein the perceptual loss corresponding to the stylized content image block is calculated based on the stylized content image block and the enlarged region in the stylized thumbnail content image, specifically comprising: Extracting a first image feature from the stylized content image block, and extracting a second image feature from the enlarged region of the stylized thumbnail content image; The difference between the first image feature and the second image feature is calculated to obtain the perceptual loss corresponding to the stylized content image block.

5. The training method of the image style transfer model according to claim 1, wherein the adversarial loss is calculated based on the stylized content image block and the style image, and specifically comprises: Comparing each stylized content image block with the style image from a style perspective, and calculating the adversarial loss corresponding to the image block; The adversarial losses corresponding to the calculated image blocks are weighted and averaged to obtain the adversarial loss corresponding to the content image.

6. A style transfer method, comprising: Determining an original content image to be processed; The original content image is segmented according to a preset segmentation rule to obtain a plurality of content image blocks with the same resolution; Inputting a plurality of content image blocks into an image style transfer model to generate a plurality of stylized content image blocks, wherein a normalization parameter in the image style transfer model is determined based on a thumbnail content image of the original content image; The multiple stylized content image blocks are spliced ​​and combined to obtain a style transferred image; wherein the image style transfer model is obtained by the training method of the image style transfer model according to any one of claims 1 to 5.

7. The style transfer method according to claim 6, further comprising: Scaling the original content image to obtain a thumbnail content image, wherein the resolution of the thumbnail content image is the same as the resolution of the content image block; Before inputting the plurality of content image blocks into the image style transfer model to generate the plurality of stylized content image blocks, the method further includes: The thumbnail content image is input into an image style transfer model, and a normalization parameter of the image style transfer model is determined when the thumbnail content image is used as input.

8. The style transfer method according to claim 7, wherein the plurality of content image blocks are input into the image style transfer model to generate a plurality of stylized content image blocks, specifically comprising: The plurality of content image blocks are input into an image style transfer model set based on the normalization parameter to generate a plurality of stylized content image blocks.

9. A training device for an image style transfer model, comprising: An acquisition module is used to obtain content images and style images from a sample set; a processing module, configured to segment the content image according to a preset segmentation rule to obtain a plurality of content image blocks of the same resolution; and to scale the content image to obtain a thumbnail content image, wherein the resolution of the thumbnail content image is the same as the resolution of the content image block; An input module, configured to input the content image block, the thumbnail content image, and the style image into a preset network model to obtain a stylized thumbnail content image and a stylized content image block; A calculation module, configured to calculate a perceptual loss based on the stylized thumbnail content image and the stylized content image block, and calculate an adversarial loss based on the stylized content image block and the style image; A training module is used to optimize and adjust the preset network model based on the perceptual loss and the adversarial loss to obtain an image style transfer model.

10. A style transfer device, comprising: A determination module, used for determining an original content image to be processed; A processing module, configured to segment the original content image according to a preset segmentation rule to obtain a plurality of content image blocks with the same resolution; A generating module, configured to input a plurality of content image blocks into an image style transfer model to generate a plurality of stylized content image blocks, wherein a normalization parameter in the image style transfer model is determined based on a thumbnail content image of the original content image; A combination module is used to splice and combine the multiple stylized content image blocks to obtain a style transfer image; wherein the image style transfer model is obtained by the training method of the image style transfer model described in any one of claims 1-5.

11. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which, when executed, cause the processor to execute the training method of the image style transfer model described in any one of claims 1 to 5 or the style transfer method described in any one of claims 6 to 8.

12. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including multiple application programs, enables the electronic device to execute the training method of the image style transfer model described in any one of claims 1-5 or the style transfer method described in any one of claims 6-8.

Images