Gating block-based diverse image style transfer method, computer device, readable storage medium and program product
By introducing gating factors in the encoder and decoder to adjust the degree of branch usage and utilizing the differentiated arrangement of convolution kernels and bottleneck layers, the problem of limited diversity of stylized images in the prior art is solved, and the generation of high-quality, diverse stylized images is realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2022-06-14
- Publication Date
- 2026-04-24
AI Technical Summary
Existing techniques struggle to achieve diverse image style transfer while maintaining the quality of stylized images, and existing methods often rely on random noise injection, resulting in limited diversity.
A diverse image style transfer method based on gating blocks is adopted. By introducing gating factors in the encoder and decoder to adjust the degree of use of independent branches, diverse stylized images are generated by utilizing different convolution kernel sizes and the differentiated arrangement of bottleneck layer residual blocks.
It achieves the generation of diverse stylized images while maintaining high quality, avoids the quality degradation caused by random noise injection, and enhances the diversity of stylized images.
Smart Images
Figure CN115293955B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of computer vision and deep learning, and in particular to a gating block-based method for diverse image style transfer, a computer device, a readable storage medium, and a program product. Background Technology
[0002] Image style transfer is a powerful image editing and artistic creation technique that can present photographs in the style of artistic images. Due to its considerable scientific and artistic value, research on this topic has attracted widespread attention in recent years. Recently, Gatys et al.'s pioneering work, "Image Style Transfer Using Convolutional Neural Networks," first proposed extracting content features and style relevance (Gram Matrix) based on the processing hierarchy of a pre-trained deep convolutional neural network (DCNN), making it possible to separate and reconstruct the content and style of any image. Although effective and flexible, it is an optimization-based method, very slow, and cannot process style transfer in real time, which limits its application. To achieve real-time stylization, feedforward style transfer methods have been proposed, which leverage the advantages of deep learning networks to incorporate computational costs into the training process. However, while enjoying inference efficiency, these methods are limited by the unnecessary requirement of training a network for each style.
[0003] Recently, general style transfer methods have been proposed, overcoming this dilemma by utilizing a single model to transfer arbitrary new styles. Similar to efficiency and generality mentioned above, diversity is another important aspect reflecting the performance of style transfer methods and has recently attracted more attention. Essentially, style transfer is a multimodal problem; a single input can correspond to multiple possible outputs. This is mainly because it is difficult to quantitatively and precisely define what the style of an artistic image is. Instead, people are encouraged to interpret the style of an artistic image from different perspectives, and there is no single correct answer. Therefore, the stylization results should be diverse, not unique, to capture the complete distribution of possible outputs. To achieve diversity in style transfer, Li et al. and Ulyanov et al., in "Diversified texture synthesis with feed-forward networks" and "Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis," used random noise vectors along with the content image as input, leveraging the variability of random noise to produce different stylization results.
[0004] However, because the content image provides strong structural prior information to the output image and has a higher dimensionality than the input noise vector, the network tends to focus more on the content image and ignore the input noise vector. To mitigate this problem, methods to reduce the similarity of different stylization results in mini-batches have been proposed. Recently, Wang et al. introduced the Deep Feature Perturbation (DFP) operation in "Diversified Arbitrary Style Transfer via Deep Feature Perturbation," which uses an orthogonal random noise matrix to perturb the deep image feature map while preserving the original style information. Although these methods have achieved significant progress, it is difficult to balance diversity and quality. Summary of the Invention
[0005] Therefore, it is necessary to provide a diverse image style transfer method based on gating blocks to address the aforementioned technical problems.
[0006] This application presents a diverse image style transfer method based on gated blocks, implemented using a diverse image style transfer network. This network includes a style generation network, which comprises an encoder and a decoder connected sequentially. The encoder is used to input the content image, and the decoder is used to output a stylized image.
[0007] The decoder includes a decoding gating block and a decoding backbone network connected in sequence. The decoding gating block includes at least a first branch and a second branch that are independent of each other but share an input. The outputs of the first branch and the second branch are transmitted to the decoding backbone network. The size of the convolution kernels of the first branch and the second branch and / or the number of residual blocks in the bottleneck layer are different.
[0008] The decoding gating block includes branches, each with a gating factor, which is used to adjust the degree of use of each branch in the decoding gating block.
[0009] Optionally, the convolution stride of the first branch and the second branch can be different.
[0010] Optionally, the number of branches in the decoding gating block is 3 to 5.
[0011] Optionally, the diversity image style transfer network includes a loss network, which is used as input to the content image, the stylized image, and the style image.
[0012] The loss network is used to compare the loss of the stylized image with that of the content image, and to compare the loss of the stylized image with that of the style image, in order to minimize the two losses and optimize the style generation network.
[0013] Optionally, the encoder is built based on a pre-trained VGG network and is also used for input style images.
[0014] Optionally, the encoder includes a coding backbone network and a coding gating block connected in sequence. The output of the coding backbone network is transmitted to the coding gating block. The coding gating block includes at least a sixth branch and a seventh branch that are independent of each other but share the same output. The size of the convolution kernels of the sixth branch and the seventh branch and / or the number of residual blocks in the bottleneck layer are different.
[0015] Optionally, the method also includes training the diverse image style transfer network, the training process of which includes:
[0016] The gating factor of any one branch of the decoding gating block is set to 1, and the gating factors of the remaining branches are set to 0.
[0017] Optionally, it also includes generating stylized images of different styles using the diverse image style transfer network, specifically including:
[0018] The gating factors of each branch included in the decoding gating block are assigned values, the gating factors range from 0 to 1, and the sum of the gating factors of each branch is 1;
[0019] Stylized images are generated using a diversity image style transfer network with gating factor assignment.
[0020] Optionally, the number of branches in the encoding gating block is 3 to 5.
[0021] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the gating block-based diversity image style transfer method described in this application.
[0022] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the gating block-based diversity image style transfer method described in this application.
[0023] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the gating block-based diversity image style transfer method described in this application.
[0024] The diverse image style transfer method based on gated blocks in this application has at least the following effects:
[0025] The first and second branches of this application can independently transmit data through the differentiated arrangement of convolution kernels and / or residual blocks in the bottleneck layer. Each branch, through the effect of the gating factor, makes the stylized image diverse and ensures the quality of the stylized image. Attached Figure Description
[0026] Figure 1 This is a schematic diagram of the structure of a diverse image style transfer network in one embodiment of this application;
[0027] Figure 2 This is a schematic diagram of the structure of a diverse image style transfer network in one embodiment of this application;
[0028] Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0029] The goal of the diversification image style transfer task is to transform everyday images into diverse stylized images based on the style of artistic images. Although existing techniques have achieved some diversity through noise injection, they still fail to address two problems: 1) relatively limited diversity, and 2) a significant decrease in quality.
[0030] See Figure 1One embodiment of this application provides a method for style transfer of diverse images based on gating blocks, which is implemented using a style transfer network. The style transfer network includes a style generation network, which includes an encoder and a decoder connected in sequence. The encoder is used to input the content image, and the decoder is used to output the stylized image.
[0031] The decoder includes a decoding gate block and a decoding backbone network connected in sequence. The decoding gate block includes at least a first branch and a second branch that are independent of each other but share the same input. The outputs of the first branch and the second branch are transmitted to the decoding backbone network. The size of the convolution kernels of the first branch and the second branch and / or the number of residual blocks in the bottleneck layer are different.
[0032] Each branch in the decoding gate block has a gating factor, which is used to adjust the degree of use of each branch in the decoding gate block.
[0033] In this embodiment, different style patterns are obtained from the same style image (artistic image) based on a network architecture with different branches. The number of branches in the decoding gate block is 3 to 5, including a first branch and a second branch. The first branch and the second branch can independently achieve data transmission through the differentiated arrangement of convolutional kernels and / or residual blocks in the bottleneck layer. Each branch, through the action of the gating factor, makes the stylized image diverse. Sampling branches control rather than inject random noise, which improves the quality of the stylized image. The decoding backbone network is the part of the decoder that has not been improved in the prior art, and will not be described in detail. The working method of the gating factor is described in the relevant content below. It can be understood that in the technical scenario of this embodiment, the value range of the gating factor is 0 to 1.
[0034] The decoding gate block includes branches that also perform upsampling, as detailed in the relevant embodiments below. The order and number of the first and second branches in the decoding gate block are not limited, and the convolution stride of the first and second branches can also be set differently.
[0035] See Figure 1 and Figure 2 Specifically, the diversity image style transfer network also includes a loss network. The loss network takes the content image, the stylized image, and the style image as input. It compares the loss of the stylized image with that of the content image, and compares the loss of the stylized image with that of the style image, minimizing both losses to optimize the style generation network. For details on the implementation of the loss network, please refer to the related embodiments below and existing technologies.
[0036] Most existing feedforward style transfer methods employ an encoder-decoder architecture, which uses an encoder E to extract semantic information from an input image and a decoder D to convert the encoded semantic feature map into a stylized image. In some existing methods, the encoder is trainable (e.g., as described by Johnson et al. in their paper "Perceptual Losses for Real-Time Style Transfer and Super-Resolution"), while in others the encoder is fixed (e.g., as described by AdaIN and SANet et al. in their paper "Arbitrary Style Transfer with Style-Attentional Networks"). The decoding gating blocks in the embodiments of this application are improvements based on existing style generation networks, such as feedforward style transfer models, and possess compatibility. The existing style generation network can be, for example, an arbitrary style transfer model or a single-style transfer model.
[0037] See Figure 1 In one embodiment, the diverse image style transfer network can be obtained by improving upon existing single-style transfer models, with an encoder that is not trainable. The encoder simultaneously inputs content and style images and aligns content and style features (e.g., ...). Figure 1 (The transformation of the symbols shown). Accordingly, the encoder may be built, for example, based on a pre-trained VGG network (e.g., VGG-19), and the encoder is also used for the input style image.
[0038] See Figure 2 In one embodiment, the diverse image style transfer network can be obtained by improving any style transfer model in the prior art, and the encoder is trainable. Accordingly, the encoder includes a coding backbone network and a coding gating block connected in sequence. The output of the coding backbone network is transmitted to the coding gating block, which includes at least a sixth branch and a seventh branch that are independent of each other but share outputs. The size of the convolutional kernels of the sixth branch and the seventh branch and / or the number of residual blocks in the bottleneck layer are different.
[0039] Each branch within the encoding gate block also has a gating factor, which is used to adjust the degree of use of each branch within the encoding gate block. The number of branches in the encoding gate block is 3 to 5, including a sixth and seventh branch. The construction method of different branches in the encoding gate block is the same as that of the decoding gate block. The difference is that the sampling of the encoding gate block is downsampling. The interaction between different branches of the encoding gate block and the coupled decoder gate block further enhances the diversity of the stylized image. It can be understood that if the number of branches in both the decoding and encoding gate blocks is too small, diversity will not be satisfied; if it is too large, it will affect the training and use of the model.
[0040] The "gating pattern pyramid" (GPP) introduced in various embodiments of this application is referred to simply as a "gating block". The gating block includes the decoding gating block D provided in various embodiments of this application. GPP (Decoder gated pyramid block) and encoder gated block E GPP (Encoder gated pyramid block), and the two are combined with the backbone network to form the decoder and encoder. Encoding backbone network E share (Encoder backbone network) Referencing the first few layers of the encoder, it learns low-level visual information from the input image. Decoding backbone network D share (Decoder backbone network) Outputs a stylized image based on the fusion result of the decoding gate blocks.
[0041] Now let's talk about decoding gate block D GPP and coding gate block E GPP The kernel size, stride size, number of residual blocks in the bottleneck layer, and sampling method are functionally implemented and analyzed.
[0042] When convolutional layers with different kernels or strides convolve different local neural blocks of a feature map, different stylistic features can be extracted from images of the same style, assigning different receptive fields to the images and affecting the stroke size of the stylized images. By fine-tuning the kernel or stride size in the network, visually plausible but perceptually different stylized results can be produced.
[0043] Many style transfer methods often incorporate bottleneck layers within the encoder and decoder to capture more image features. Bottleneck layers typically contain numerous residual blocks. Intuitively, the more residual blocks, the more style information the bottleneck layer acquires. Using fewer residual blocks, the network focuses on salient style patterns, producing clean and neat stylization results; with more residual blocks, the network learns more refined style patterns, producing richer and more vibrant stylization results. Despite the differences, all these stylization results represent reasonable and satisfactory style transfer solutions.
[0044] In deep learning networks, a feature map can be downsampled using strided convolution, max pooling, or average pooling. Specifically, strided convolution downsamples the feature map by using a learnable convolutional kernel, while max pooling or average pooling downsamples it by summing the maximum or average value in the feature map. Due to their different characteristics, their stylization results exhibit different color and texture styles.
[0045] Similar to downsampling, deep learning networks employ various methods to upsample feature maps, such as deconvolution, nearest neighbor interpolation, and bilinear interpolation. Specifically, deconvolution upsamples the feature map by using a learnable deconvolution kernel; nearest neighbor interpolation selects the value of the nearest point to fill in missing locations; and bilinear interpolation uses the distance-weighted average of the four nearest points to fill in missing locations. Based on the different characteristics of these upsampling methods, we obtain different stylization results.
[0046] Next, we will explain the diversity image style transfer network in detail using the formula.
[0047] See Figure 2 In one embodiment, the encoder E sequentially includes an encoding backbone network E share and coding gate block E GPP Encoded gate block E GPP It consists of n branches with different network architectures, where n can be 3 to 5. For example, it can include the first branch and the second branch mentioned above, used to obtain different high-level image features. ,
[0048]
[0049] in:
[0050] Represents a content image;
[0051] Indicates the coding backbone network E share Input content image;
[0052] E represents GPP The i-th branch in the process E share The output results are used to obtain high-level image features. .
[0053] In order to provide encoding gate block E GPP The control of each branch in the process involves introducing a gating function G for each branch:
[0054]
[0055]
[0056] in,
[0057] express One constraint is that the value ranges from zero to one;
[0058] High-level image features representing the output of the encoder gate block;
[0059] This represents the gating factor, which is used to adjust the degree of utilization of each branch in the encoder gating block;
[0060] This indicates the superposition result of using different branches in the encoder gating block;
[0061] F E This is the output of encoder E.
[0062] See Figure 1 and Figure 2 Decoder D includes, in sequence, decoding gate block D. GPP and decoding backbone network D share Decoding gate block D GPP It consists of m branches with different architectures to learn different style patterns. ,
[0063]
[0064] in:
[0065] F E This indicates the output of encoder E;
[0066] Indicates decoding gate block D GPP The i-th branch in;
[0067] This represents the style pattern of the i-th branch.
[0068] The value of m can be, for example, 3 to 5.
[0069] Decoding gate block D GPP To produce style pattern fusion results Specifically, it is accomplished using the following formula:
[0070]
[0071] in, This represents the style pattern of the i-th branch. This represents the result of merging different branch style patterns.
[0072] Decoding backbone network D share Used to enhance the fusion results to produce more reasonable stylized images. .
[0073]
[0074] in, D represents the decoding backbone network. share Output a stylized image based on the fusion result of the style patterns.
[0075] In order to provide decoding gate block D GPP The control of each branch involves introducing a gating function G for each branch. This can be understood as the decoding gating block D... GPP Gating functions and encoded gating blocks E GPP The gating functions are consistent:
[0076]
[0077]
[0078] in,
[0079] express One constraint is that the value ranges from zero to one;
[0080] Represents high-level image features output by the decoding gate block;
[0081] This represents the gating factor, which is used to adjust the degree of utilization of each branch in the decoding gating block;
[0082] This represents the superposition result of using different branches in the decoding gate block;
[0083] F E This is the output of the decoding gate block.
[0084] In one embodiment, the method further includes training a diverse image style transfer network, wherein the training process includes: assigning a gating factor of 1 to any branch of the decoding gating block and assigning a gating factor of 0 to the gating factors of the remaining branches, until training is completed.
[0085] In one embodiment, the method includes generating stylized images of different styles using a diverse image style transfer network. Specifically, this includes: assigning values to the gating factors of each branch of the decoding gating block, wherein the values of the gating factors range from 0 to 1, and the sum of the gating factors of each branch is 1; and generating stylized images using the diverse image style transfer network after assigning the gating factors.
[0086] Understandably, during the training phase, It is set to 0 or 1, meaning the branches of the gating blocks are trained independently to prevent them from interfering with each other. During the usage phase, These can be fractions, allowing the outputs of these branches to be blended in any proportion to produce more diverse results. Experiments show that the diversity is not achieved through random initialization of the diversity image style transfer network at the start of each training iteration, but rather through the decoding and / or encoding gates provided in each embodiment.
[0087] Based on the experimental process and results, the schemes of each embodiment of this application are further explained. The methods of each embodiment are applied to three existing feedforward style transfer models, including Johnson et al., AdaIN, and SANet (see the reference information above). The gating blocks proposed in each embodiment enhance the original network architecture while preserving the original objective function and hyperparameters.
[0088] Different branches in the gated pyramid block, which are built on the backbone of the original encoder and / or decoder, differ in kernel size, stride size, number of residual blocks in the bottleneck layer, downsampling or upsampling.
[0089] In all experiments, we set the hyperparameter (number of branches) m = 5. For the gating factor... In each iteration of the training phase, a branch is randomly selected, and its gating weight is set. = 1 (other settings are 0). Sampling from a uniform distribution. The value is then sampled and normalized (e.g., by L1 specification) to satisfy the constraints. .
[0090] The loss network can be, for example, a pre-trained VGG-19 network, which is widely used in existing style transfer methods. Content images can be from the MS-COCO dataset, and style images can be from the WikiArt dataset. During training, each image is first resized to 512 pixels using the original image scale, and then a region of size 256 × 256 is randomly cropped. The input image can be of any size.
[0091] Experimental Performance Evaluation: To demonstrate the effectiveness of the Gated Pattern Pyramid Block (GPP) in enhancing the diversity of stylization results, the proposed method is compared with a large number of state-of-the-art qualitative and quantitative evaluations. A comprehensive ablation study is also conducted to show how each component affects the final result. To explore the effect of the proposed GPP, we apply it to three existing feedforward style transfer methods (i.e., Johnson et al., AdaIN, and SANet). Our GPP successfully extends the original style transfer methods to produce significantly different stylization results. We observe that water in the different stylized images exhibits different colors and textures.
[0092] For quality, we used a user survey to compare the stylization results of each method with and without GPP blocks. For each pair of methods, we generated 20 pairs of stylized images. Fifteen pairs were randomly selected for each target, and users chose the pair with the best visual impact. We ultimately collected 600 valid votes for each pair of methods from the Amazon Mechanical Turk (AMT) platform.
[0093] For simplicity, we default to using Johnson et al. + GPP for our ablation study, which is similar to AdaIN + GPP and SANet + GPP. We propose setting the hyperparameter i in Equation (2) to 0 or 1 during the training phase, instead of random scores. To verify the effectiveness of this setting, we conducted an ablation study, which showed that the setting adopted in this paper achieved a significant gain in diversity. The reason may be that setting i to 0 or 1 prevents different branches from interfering with each other during training, which helps them learn different style patterns independently. When m=1, as expected, the stylized images have no diversity. When we amplify m with multiple network branches, different style patterns are learned, and the stylization results become more and more diverse. However, it is worth noting that the growth in diversity tends to weaken after m becomes sufficiently large. On the other hand, the larger m is, the longer the running time t of the method becomes. Finally, we decided to use m=5 as a tradeoff between diversity and efficiency. The original style transfer method (m=1) and the corresponding diversified style transfer method (m=5) have similar visual quality, and the number of branches m does not affect the quality of the stylized images.
[0094] The location where GPP blocks are applied. Recent style transfer methods typically employ an encoder-decoder architecture to synthesize artistic images. For some methods (e.g., AdaIN and SANet), since the encoder is a pre-trained VGG-19 network, GPP blocks can only be applied to the decoder. For others (e.g., Johnson et al.), GPP blocks can be applied to: (a) the encoder, (b) the decoder, or (c) both. To determine which case the model achieves optimal performance, we observed that case (c) (applying GPP to both the encoder and decoder) yielded the best subset, followed by case (b) (applying GPP only to the decoder), while case (a) (applying GPP only to the encoder) performed the worst. This also explains why GPP can generate a wider variety of images compared to AdaIN+GPP and SANet+GPP.
[0095] Each embodiment of this application utilizes gating blocks, producing significantly different stylization results compared to many existing feedforward style transfer methods without sacrificing visual quality. After a detailed analysis of the factors influencing the color and texture patterns of stylized images in the style generation network, each embodiment introduces decoding gating blocks, combining them with a decoding backbone network to form a new decoder, compared to traditional decoders. The new encoder extracts different style patterns from the same artistic image, producing diverse and high-quality stylization results. The gating block-based diverse image style transfer methods provided by each embodiment are compatible with many feedforward style transfer models and endow them with the ability to generate different stylization results. Compared to the existing method of introducing random noise to achieve diversity, this provides a new perspective for multimodal transfer research, demonstrating superiority and effectiveness.
[0096] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data for a diversity image style transfer network. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a gated block-based diversity image style transfer method.
[0097] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0098] A gated block-based style transfer method for diverse images is implemented using a diverse image style transfer network. The diverse image style transfer network includes a style generation network, which comprises an encoder and a decoder connected sequentially. The encoder is used as the input content image, and the decoder is used as the output stylized image.
[0099] The decoder includes a decoding gate block and a decoding backbone network connected in sequence. The decoding gate block includes at least a first branch and a second branch that are independent of each other but share the same input. The outputs of the first branch and the second branch are transmitted to the decoding backbone network. The size of the convolution kernels of the first branch and the second branch and / or the number of residual blocks in the bottleneck layer are different.
[0100] Each branch in the decoding gate block has a gating factor, which is used to adjust the degree of use of each branch in the decoding gate block.
[0101] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0102] A diverse image style transfer method based on gated blocks is implemented using a diverse image style transfer network. The diverse image style transfer network includes a style generation network, which comprises an encoder and a decoder connected in sequence. The encoder is used to input the content image, and the decoder is used to output the stylized image. The decoder includes a decoding gated block and a decoding backbone network connected in sequence. The decoding gated block includes at least a first branch and a second branch that are independent of each other but share the same input. The outputs of the first branch and the second branch are transmitted to the decoding backbone network. The size of the convolutional kernels of the first branch and the second branch and / or the number of residual blocks in the bottleneck layer are different. Each branch in the decoding gated block has a gating factor, which is used to adjust the degree of use of each branch in the decoding gated block.
[0103] In one embodiment, a computer program product is provided, including computer instructions that, when executed by a processor, perform the following steps:
[0104] A diverse image style transfer method based on gated blocks is implemented using a diverse image style transfer network. The diverse image style transfer network includes a style generation network, which comprises an encoder and a decoder connected in sequence. The encoder is used to input the content image, and the decoder is used to output the stylized image. The decoder includes a decoding gated block and a decoding backbone network connected in sequence. The decoding gated block includes at least a first branch and a second branch that are independent of each other but share the same input. The outputs of the first branch and the second branch are transmitted to the decoding backbone network. The size of the convolutional kernels of the first branch and the second branch and / or the number of residual blocks in the bottleneck layer are different. Each branch in the decoding gated block has a gating factor, which is used to adjust the degree of use of each branch in the decoding gated block.
[0105] In this embodiment, the computer program product includes a program code portion for performing the steps of the gating block-based diversity image style transfer method of the various embodiments of this application when the computer program product is executed by one or more computing devices. The computer program product may be stored on a computer-readable recording medium. The computer program product may also be provided for download via a data network (e.g., via RAN, via the Internet, and / or via RBS). Alternatively or additionally, the method may be encoded in a field-programmable gate array (FPGA) and / or application-specific integrated circuit (ASIC), or the functionality may be provided for download by means of a hardware description language.
[0106] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0107] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered to be within the scope of this specification. When technical features of different embodiments are embodied in the same drawing, it can be regarded as the drawing also disclosing examples of combinations of the various embodiments involved.
[0108] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A diverse image style transfer method based on gated blocks, implemented using a diverse image style transfer network, wherein the diverse image style transfer network includes a style generation network, the style generation network including an encoder and a decoder connected in sequence, the encoder being used as input content images, and the decoder being used as output stylized images, characterized in that, The decoder includes a decoding gating block and a decoding backbone network connected in sequence. The decoding gating block includes at least a first branch and a second branch that are independent of each other but share an input. The outputs of the first branch and the second branch are transmitted to the decoding backbone network. The size of the convolution kernels of the first branch and the second branch and / or the number of residual blocks in the bottleneck layer are different. The decoding gate block includes branches, each with a gating factor, which is used to adjust the degree of use of each branch in the decoding gate block.
2. The method for diverse image style transfer according to claim 1, characterized in that, The convolution stride of the first branch and the second branch are different.
3. The method for diverse image style transfer according to claim 1, characterized in that, The diverse image style transfer network includes a loss network, which is used as input to the content image, the stylized image, and the style image. The loss network is used to compare the loss of the stylized image with that of the content image, and to compare the loss of the stylized image with that of the style image, in order to minimize the two losses and optimize the style generation network.
4. The method for diverse image style transfer according to claim 1, characterized in that, The encoder is built on a pre-trained VGG network and is also used to input style images.
5. The method for diverse image style transfer according to claim 1, characterized in that, The encoder includes a coding backbone network and a coding gating block connected in sequence. The output of the coding backbone network is transmitted to the coding gating block. The coding gating block includes at least a sixth branch and a seventh branch that are independent of each other but share the same output. The size of the convolution kernels of the sixth branch and the seventh branch and / or the number of residual blocks in the bottleneck layer are different.
6. The method for diverse image style transfer according to claim 1, characterized in that, It also includes training the diverse image style transfer network, the training process of which includes: The gating factor of any one branch of the decoding gating block is set to 1, and the gating factors of the remaining branches are set to 0.
7. The method for diverse image style transfer according to claim 1, characterized in that, It also includes generating stylized images of different styles using the aforementioned diverse image style transfer network, specifically including: The gating factors of each branch included in the decoding gating block are assigned values, the gating factors range from 0 to 1, and the sum of the gating factors of each branch is 1; Stylized images are generated using a diversity image style transfer network with gating factor assignment.
8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the gating block-based diversity image style transfer method according to any one of claims 1 to 7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the gating block-based diversity image style transfer method as described in any one of claims 1 to 7.
10. A computer program product, including computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the gating block-based diversity image style transfer method according to any one of claims 1 to 7.