Style transfer system, method, and device based on attention recurrent adversarial network

Abstract

The present invention discloses a style transfer system, method, and device based on an attention-based recurrent adversarial network. The method first selects two images A and B of different styles and inputs them into the network; randomly nests and crops multiple small blocks of image A and inputs them into a multi-scale block transformer encoder to learn features, fuses low-dimensional global information layer by layer through deconvolution and upsampling, and finally fuses high-dimensional global information after dynamic filtering to generate a transfer result; simultaneously trains the A2B and B2A mappings through a recurrent adversarial approach; introduces a block-based discriminator and content loss function for training convergence, and finally completes the mutual transfer of styles of A and B. The present invention proposes for the first time a zero-shot learning style transfer method based on a recurrent adversarial network, which utilizes nested cropping of small blocks to mine the relationship features within and between blocks, and can generate a realistic and reasonable transfer effect. It is superior to existing methods and has the advantages of strong versatility, low data dependence, and strong personalization of style generation.

Description

Technical Field

[0001] The present invention relates to the fields of computer graphics and deep learning, and in particular to a style transfer system, method, and device based on an attention recurrent adversarial network. Background Art

[0002] Image-to-image translation is a long-standing problem that seeks to transfer an image to the style of a reference image. While generative adversarial networks have achieved great success in generating high-quality results, most of them rely on a database of images with the same style, making them prohibitively expensive for practical applications. Recent work has demonstrated that modeling the internal statistical information of small patches contained in user-provided images is effective for style transfer tasks. This solution can handle a wide variety of style images provided by ordinary users who lack access to large datasets, as these inputs are not seen by the network before training, making them more suitable for real-world applications. While existing research has achieved significant success with zero-shot or one-shot pre-processing, the authenticity of their generated results is limited by the deep feature representation capabilities of the learned convolutional kernels. Due to the limited field of view of the local receptive field and the shared parameters of the convolutional kernels used to encode spatial correlations between pixels, these methods may produce stylized results that are not what the user intended when the reference style image depicts different regions of the scene with different style patterns. Therefore, a deep model that relies on minimal prior knowledge of the signal structure is needed to improve the quality of stylization. Inspired by recent advances in natural language processing, transformer-based models have recently demonstrated a powerful ability to encode the rich relationships present in input signals for computer vision and image processing tasks. Their self-attention mechanism allows for capturing dependencies between distant and nearby elements of the input sequence, which is equally well-suited for exploring internal relationships within sub-patch images of varying styles. This method proposes a novel multi-scale block transformer encoder that fully exploits the relationships between all elements in randomly cropped sub-patch-level features for zero-shot learning of image translation and stylization tasks. Another obstacle to practical applications of many existing image translation methods is their requirement for paired training data. To overcome this limitation, CycleGAN proposes to avoid the laborious task of constructing paired datasets. Its cycle-consistency learning scheme has been successfully applied to many tasks, such as makeup transfer and medical image processing. However, CycleGAN requires large datasets to learn bidirectional style mappings. A recent work on video style transfer has addressed this issue, using only a small number of key images for training. However, it primarily focuses on transferring the style drawn by a user on a key frame to the same object in different consecutive frames of a video. It is not suitable for transferring style between images containing different objects.

[0003] Current methods require learning the characteristic distribution of a given style from a large amount of data, thus requiring a large number of input images of similar styles. Furthermore, since images of similar styles often exhibit significant variation, training with large amounts of data weakens the individual styles, resulting in a bias towards an average style pattern. To mitigate the difficulties encountered by users in finding large datasets and paired images of different styles, this paper introduces a cycle-consistency learning scheme that simultaneously optimizes using a multi-scale block-based approach. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a style transfer system, method and device based on an attention recurrent adversarial network.

[0005] To achieve the above-mentioned object, the technical solution of the present invention is: a style transfer system based on an attention recurrent adversarial network, comprising two sets of nested random cropping modules for images and a recurrent adversarial generation module; the recurrent adversarial generation module comprises two sets of generators and corresponding discriminators; the generator comprises a multi-scale block transformer encoder module and a dynamic filtering decoder module;

[0006] The image nested random cropping module is used to randomly crop each image into multiple nested sub-image blocks;

[0007] The multi-scale block transformer encoder module includes a block-level transformer encoder and a cross-block-level transformer encoder, which encodes the sub-blocks obtained by the image nested random cropping module to obtain high-dimensional features;

[0008] The dynamic filtering decoder module is used to deconvolve the high-dimensional features obtained by the multi-scale block transformer encoder module, dynamically filter the shallow high-dimensional features in the deconvolution stage, and adaptively learn the required features to reduce the influence of the original features;

[0009] The cycle adversarial generation module uses adversarial loss, cycle consistency loss and reconstruction loss to constrain the training of bidirectional mapping to complete the mutual transfer of image styles.

[0010] The present invention proposes a style transfer method based on an attention recurrent adversarial network applied to the above system, comprising the following steps:

[0011] S1: Select two training images A and B of different styles, where A belongs to the x domain and B belongs to the y domain. Use the image nested random cropping module to perform data preprocessing. In each iteration, nested cropping is performed on each image at a random position to obtain multiple nested sub-block combinations of specified sizes.

[0012] S2, the preprocessed data obtained in step S1 is reduced in dimension by convolution and the number of channels is increased to obtain block-based convolution features, which are then input into the multi-scale block transformer encoder module for encoding to obtain high-dimensional features and learn the self-attention information within the block and the self-attention information between blocks;

[0013] S3, using the dynamic filtering decoder module to process the high-dimensional features t obtained in step S2 c Deconvolution is performed, and global content information is integrated layer by layer through deconvolution and upsampling. In the last layer, a dynamic filter is used to filter the high-dimensional features of the global shallow layer, and the required shallow content global features are adaptively learned to obtain the image generation result.

[0014] S4, based on the cycle-consistent adversarial method, simultaneously trains two sets of generators, uses their corresponding discriminator adversarial loss, and then uses the pixel-level L2 paradigm to calculate the cycle-consistent loss and reconstruction loss. The generator training is constrained by the above three losses until the generator converges, and finally completes the mutual style transfer of training images A and B.

[0015] A style transfer device based on an attention recurrent adversarial network includes one or more processors for use in the above-mentioned style transfer method based on an attention recurrent adversarial network.

[0016] A computer-readable storage medium stores a program thereon, which, when executed by a processor, is used to implement the above-mentioned style transfer method based on the attention recurrent adversarial network.

[0017] The present invention provides the following beneficial effects: The proposed style transfer system based on an attention-based recurrent adversarial network utilizes a multi-scale block transformer encoder, eliminating the need for pre-training models with large datasets or labeling of real target data. Style transfer can be achieved using only two images through a recurrent adversarial training scheme. The proposed method proposes a recurrent adversarial transformer encoder and a deep model based on a Generative Adversarial Network (GAN) to address image-to-image translation using a zero-shot learning approach. The network presented in this method minimizes user requirements for image stylization applications and outperforms existing methods in zero-shot image translation tasks. The proposed method only requires two images to generate high-quality, style-appropriate transfer results. However, due to the limited information content of a single image, it is difficult for a generative network to train a robust transfer network based on this information. The proposed method utilizes multi-scale blocks for data augmentation, deeply exploring the image's style information and structural distribution. The multi-scale block transformer learns intra- and inter-block relational features, training the mapping between content and style of the two images using a cycle-consistent scheme. The proposed cycle-consistent scheme, based on cycle-consistent error and reconstruction error, addresses the lack of real-world target results in style transfer tasks, generating transfer results with a reasonably distributed target style. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the style transfer method based on a single image of the present invention;

[0019] Figure 2 It is a flow chart of the style transfer method based on generative adversarial network and transformer for multi-scale block zero-shot learning of the present invention;

[0020] Figure 3 is a schematic diagram of the method of the present invention in a multi-scale block transformer encoder;

[0021] Figure 4 is the result of image translation in different scenarios by the method of the present invention;

[0022] Figure 5 It is the result of the method of the present invention on different style reference images on consecutive frames of the video;

[0023] Figure 6 1 is a schematic diagram comparing the migration results of the method of the present invention with those of the existing methods when the input styles are different painting images;

[0024] Figure 7 3. It is a schematic diagram comparing the migration results of the method of the present invention with those of the existing method in a scenario where the input style is a line drawing image;

[0025] Figure 8Schematic diagram of the results of the ablation experiment on important network modules of the present invention;

[0026] Figure 9 It is a schematic diagram of the results of the ablation experiment on the multi-scale block transformer of the present invention;

[0027] Figure 10 Schematic diagram of the results of the ablation experiment on the selection of convolution kernel combinations for the dynamic filter module of the present invention;

[0028] Figure 11 This is a schematic diagram of the result of the number of nested clipping blocks in the image preprocessing stage of the present invention;

[0029] Figure 12 It is a schematic diagram of the result of nested clipping of the size of small blocks in the image preprocessing stage of the present invention. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the following, in conjunction with the accompanying drawings, further describes in detail the style transfer system and method based on the attention recurrent adversarial network proposed in the present invention. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not intended to limit the present invention.

[0031] Given the limitations of existing methods, this paper aims to reduce the need for large datasets by leveraging multi-scale blocks, while simultaneously learning relationships within and between blocks using a transformer encoder. Given two images x and y of different styles, the goal is to transfer their styles while preserving their individual content. Because the present invention's training process relies on only two images, to provide the extensive information required for the transfer task, the present invention utilizes a block-based technique that is sufficient to describe style-related features. After extracting features from nested sub-blocks randomly sampled from the input image using convolutional layers, the present invention learns not only self-attention within the sub-blocks but also attention between sub-blocks of different scales. The multi-scale block transformer encoder can exploit all possible relationships within and between sampled blocks to describe the style information carried by the input image. Since no learned style is assumed, the present network needs to handle the vast diversity of image styles. Therefore, a dynamic filtering module is integrated into the decoder to adaptively learn filters for different styles when synthesizing the final result. Furthermore, to avoid being trapped in certain local style patterns, the embeddings learned by the multi-scale block transformer encoder are combined with global features before interpretation at the decoding layer. When training the model, a cycle consistency learning scheme is used to optimize two mappings that share the same architecture, where mapping G aims to convert the style of y to the style of x, and mapping F aims to convert the style of y to the style of x. For each mapping, the image carrying the target style is called the style image, and the image to be converted is called the content image.

[0032] The present invention discloses a style transfer system based on an attention recurrent adversarial network, comprising two sets of image nested random cropping modules and a recurrent adversarial generation module; the recurrent adversarial generation module comprises two sets of generators and corresponding discriminators; the generator comprises a multi-scale block transformer encoder module and a dynamic filtering decoder module.

[0033] The nested random cropping module is used to randomly crop each image into multiple nested sub-tiles. Specifically, in the nested random cropping module, two images of different styles selected by the user are input into the network. In each iteration, each image is randomly cropped into multiple nested sub-tiles. The randomly selected sub-tiles can focus on the style information of the entire image.

[0034] The multi-scale block transformer encoder module includes a block-level transformer encoder and a cross-block-level transformer encoder; the sub-blocks obtained by the image nested random cropping module are encoded to obtain high-dimensional features, that is, the relationship features between the sub-blocks obtained by the image nested random cropping module and the blocks are learned; specifically, in the multi-scale block transformer encoder module, the obtained pre-processed data is reduced in dimension and the number of channels is increased through convolution; the convolved block features are input into the multi-scale block transformer encoder to learn the self-attention information within the block and the self-attention information between the blocks.

[0035] The structure of the multi-scale block transformer encoder module used in the system of the present invention is as follows Figure 3 As shown in Figure 1, it consists of M=2 blocks, which include a multi-head self-attention module and a feed-forward MLP layer. The positional encoding mechanism of the transformer enhances its ability to exploit the relationship between components and takes into account their relative positions. However, a side effect of positional encoding is that if only the cropped sub-blocks are provided to the transformer encoder, the learned sub-block mapping will be too strong to directly map the source image to the target image. To alleviate this problem, the present invention also combines global features of different dimensions for decoding, because global features provide the relationship between the cropped sub-blocks and the entire image and weaken the irrelevant relationships within the sub-blocks. Among them, the present invention does not include global features in the multi-scale block transformer encoder because the multi-scale block transformer encoder is proposed mainly to study sub-block style-related features. If the global features are also placed in the transformer, the learning process of the multi-scale block transformer encoder will focus on the relationship between the sub-blocks and the fixed global structural features, resulting in blurred results due to the lack of attention to style details.

[0036] The dynamic filtering decoder module deconvolves the high-dimensional features obtained by the multi-scale block transformer encoder module, fuses global information layer by layer through deconvolution and upsampling, dynamically filters shallow high-dimensional features in the deconvolution stage, and adaptively learns the required features to reduce the influence of the original features; in the dynamic filtering decoder module, the global features of the original image are fused layer by layer through deconvolution and upsampling, and a dynamic filter is embedded in the last layer to filter the global shallow features, and the required global features are adaptively learned to prevent the influence of the original features from being too strong.

[0037] The cyclic adversarial generation module includes a discriminator and a generator. It uses adversarial loss, cycle-consistent loss, and reconstruction loss constraints to train a bidirectional mapping, achieving mutual style transfer between images. In this cyclic adversarial generation module, the bidirectional mapping is trained simultaneously until convergence based on a cycle-consistent adversarial scheme using adversarial loss, cycle-consistent loss, and reconstruction loss constraints, ultimately achieving mutual style transfer between images A and B.

[0038] This paper uses a multi-scale block transformer encoder to extract features based on self-attention and cross-layer attention information from randomly cropped nested patches. A dynamic filter decoder is used to interpret block-level and global features to generate the results. Two training images A and B of different styles are each assigned to a generator. The two generators share the same architecture and are trained together with their corresponding discriminators in a cycle-consistent adversarial manner.

[0039] Specifically, the present invention discloses a style transfer method based on attention loop adversarial network. Figure 2 As shown, Figure 2 The migration process of the method of the present invention is demonstrated for a given set of migration images. Two images A and B of different styles are selected for the style migration task; first, multiple small blocks are randomly nested and cropped from the two images; the cropped small blocks are combined and input into the multi-scale block transformer encoder to learn features, and low-dimensional global information is fused layer by layer through deconvolution and upsampling, and finally the high-dimensional global information after dynamic filtering is fused to generate the migration result; through a cyclic adversarial method, the mapping of A2B and B2A is trained simultaneously; a block-based discriminator and content loss function are introduced for training convergence, and finally the mutual migration of styles of A and B is completed. The method of the present invention can generate reasonable style migration results using only two images without the need for a large dataset to pre-train the model.

[0040] The method of the present invention specifically comprises the following steps:

[0041] Step S1: Select two training images A and B of different styles, where A belongs to the x domain and B belongs to the y domain. Use the image nested random cropping module for data preprocessing. In each iteration, nested crops are performed on each image at random locations to obtain multiple nested sub-block combinations of specified sizes.

[0042] In the data preprocessing stage, the image needs to be randomly cropped into K nested sub-blocks as the input of the multi-scale block transformer encoder module. The user inputs two training images A and B of different styles as the target image for style exchange. In each iteration, image A is randomly nested and cropped to obtain a specified K block combination with a specified side length pixel size (in this embodiment of the present invention, K is set to 4, and the side length pixel size is selected as {4, 8, 16, 32}). The K cropped sub-blocks are reshaped into an image with a side length of 256 pixels. (Similarly, we get the sub-block combination of image B ). The following will be based on For example, Same reason.

[0043] Step S2: The preprocessed data obtained in step S1 is subjected to convolution to reduce the dimension and increase the number of channels to obtain block features, which are then input into the multi-scale block transformer encoder module for encoding to obtain high-dimensional features and learn the self-attention information within the block and the self-attention information between blocks. Specifically, the following steps are included:

[0044] S201, the pre-processed data obtained in step S1 By reducing the dimension through convolution and increasing the number of channels, we can obtain block features of size h*w*c (K=1, 2, 3, 4) (in the embodiment of the present invention, h=w=32, c=64), and then expand it into a feature sequence of dimension C with a length of h*w.

[0045] S202: Send the feature sequence expanded in step S201 to the block-level transformer encoder T p Encode in and learn the sub-block level relationship features (K=1,2,3,4). Then, the sub-block level relation features are concatenated in the channel dimension to generate features of size (h,w,Kc) The features Input cross-block level transformer encoder T c To obtain high-dimensional features, that is, the relationship features t across multi-scale blocks c .

[0046] Step S3: Use the dynamic filtering decoder module to process the high-dimensional features t obtained in step S2 c Deconvolution is performed. Through deconvolution and upsampling, global content information is integrated layer by layer. In the last layer, a dynamic filter is used to filter the high-dimensional features of the global shallow layer. The required shallow content global features are adaptively learned to prevent the influence of the original content features from being too strong. Specifically:

[0047] Most existing style transfer techniques rely on conventional CNN layers to decode features. However, their fixed filters cannot handle the diversity of style features and therefore cannot adaptively infer different image styles. To address this issue, this paper introduces dynamic convolution to filter shallow global networks.

[0048] Specifically: the relationship feature t learned in step S3 c Perform deconvolution, and fuse the global content features of the corresponding dimensions layer by layer through deconvolution and upsampling (d=64,128), to increase the feature dimension. To avoid the influence of shallow features being too strong, the shallow global content features are not directly fused after upsampling the last layer. (d=256), but a dynamic filter is embedded in the network to dynamically filter the shallow global content features, and then fuse the filtered global features with the upsampled features. The dynamic filter performs convolution operations on convolution kernels of different sizes according to different inputs. k Calculate the attention weight Π k (x), Input each dynamic convolution submodule for convolution and multiply by the weight π k (x), after mini-batch normalization and relu activation, the adaptive global shallow features are obtained Then upsampling is performed again, and finally multi-layer convolution is used to reduce the number of channel bits and deepen the network to obtain a 256*256*3 network input, which is the image generation result.

[0049] For training image A, after steps S1-3 (i.e., generator G), the generated image fake_B is obtained. Similarly, for training image B, after simultaneously performing steps S1-3 (i.e., generator F), the generated image fake_A is obtained. Generator G: x→y aims to transform image A in the x domain into image fake_B in the y domain. Generator F: y→x aims to transform image B in the y domain into image fake_A in the x domain. The x and y domains represent two different style domains.

[0050] Step S4: Based on the cycle-consistent adversarial method, the bidirectional mapping generator G and the generator F are trained simultaneously until convergence. The adversarial loss is calculated through their corresponding discriminators. The cycle-consistent loss and reconstruction loss are calculated using the pixel-level L2 norm. By constraining the network training with the above three losses, the style transfer between the training images A and B is finally completed. The specific sub-steps are as follows:

[0051] S401, through the discriminator D y , D x Calculate the adversarial loss for the generated results fake_B and fake_A obtained in step S3, that is, constrain fake_B and B (fake_A and A) to belong to the same style domain y(x). Block-based discriminator D y The training objective function is:

[0052]

[0053] Where, x(p k )~P data [x(p)], represents the sampled data from the data distribution P dataSub-block of [x(p)], P data [x(p)] represents the data distribution of image patches with the same style.

[0054] S402, use cycle consistency loss to further constrain the training of the two generators G, F and the corresponding discriminator to avoid mode collapse, specifically:

[0055] The adversarial loss is sufficient to generate reasonable images in the target domain, but it cannot ensure the accuracy and diversity of the results (mode collapse). Therefore, the cycle consistency loss is introduced into the network. First, the generator F generates the image fake_B generated by the generator G back to itself, that is, to generate rec_A; similarly, the generator G generates the image fake_A generated by the generator F back to itself, that is, to generate rec_B. The cycle consistency loss aims to measure the pixel-level difference loss between rec_A and the A image through the L1 norm. At the same time, the cycle consistency loss also includes the L1 loss of rec_B and B. Its training objective function is expressed as follows:

[0056]

[0057] In S403, to maintain the content of the original image, the network also introduces a reconstruction loss to constrain reverse generation. Because the goal of the generator G is to transform any input image into the style domain of y, the input image B needs to be able to be reconstructed back into image B itself, and the same is true for the mapping F.

[0058] The reconstruction loss function is defined as follows:

[0059]

[0060] S404, the objective function is obtained by the above three loss constraints:

[0061] L(G, F, D x , D y )=L GAN (G, D y ,x,y)+L GAN (F, D x , y, x)+λ idt L idt (G, F, x, y) + λ cyc L cyc (G, F, x, y) (4)

[0062] Among them, λ idt and λ cyc Controls the weight between cycle consistency and reconstruction accuracy.

[0063] After multiple rounds of iteration, the above steps S1 to S4 are repeated until the generators G and F converge, and the target results fake_A and fake_B are finally obtained through iteration.

[0064] Corresponding to the aforementioned embodiment of the style transfer method based on the attention recurrent adversarial network, the present invention also provides an embodiment of the style transfer device based on the attention recurrent adversarial network.

[0065] An embodiment of the present invention provides a style transfer device based on an attention recurrent adversarial network, including one or more processors for implementing the style transfer method based on the attention recurrent adversarial network in the above embodiment.

[0066] The embodiment of the style transfer device based on the attention recurrent adversarial network of the present invention can be applied to any device with data processing capabilities, and the any device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, in addition to the processor, memory, network interface, and non-volatile memory, the any device with data processing capabilities in which the device in the embodiment is located can also include other hardware according to the actual functions of the any device with data processing capabilities, which will not be described in detail.

[0067] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0068] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0069] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the style transfer method based on the attention recurrent adversarial network in the above embodiment is implemented.

[0070] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0071] The method of the present invention does not require a large dataset for training, as it only uses two images as the style image and content image for the bidirectional mapping, which facilitates data collection. Most of the results presented in the experiments were generated using images randomly downloaded from the CycleGAN dataset and image search engines. In this specific embodiment, the images are first resized to 256*256 to strike a balance between resolution and GPU memory usage, as the fully connected transformer architecture consumes a lot of memory. In this specific embodiment, four nested sub-blocks are cropped in each iteration in a nested manner, where the larger sub-block always contains the smaller sub-blocks. The size of the nested sub-blocks does not significantly affect the quality of the final result. However, if the style-related features have a finer granularity, the size of the smallest sub-block can be smaller. In this specific embodiment, the four sub-block sizes are fixed to 4, 8, 16, and 32, respectively. All cropped sub-blocks are first resized to 256 before being fed into the convolutional layer to obtain feature maps. Sub-block-based self-attention and cross-level attention information are then learned through a multi-scale block transformer encoder. In the decoding stage of the generator, the kernel sizes of the three dynamic convolution blocks are 1, 3, and 5 respectively. When processing real-life photo translation, if the features generated by the dynamic filter are connected with the global features of the original input, better clarity can be produced. Therefore, the present invention adds an option to control whether to pass the shallow features through the dynamic filter. Although the required shallow features can be adaptively learned through the dynamic filter to obtain more personalized stylized pictures. However, in the image translation task between real images, this network will directly connect the shallow features (that is, the shallow features do not pass through the dynamic filter) to obtain clearer image results. The method of the present invention trains the generator through adversarial loss, consistency loss, and reconstruction loss. They are trained through λ cyc ,λ idt ,λ gan and λ contentWeighted to balance the content preservation and stylization richness. In a specific embodiment of the invention, cyc and λ idt Set to 100 for some and 1 for others.

[0072] In order to verify the effects of the present invention, the present invention has conducted various comparative experiments in various scenarios.

[0073] Figure 4 We demonstrate the ability of our method to perform image translation between real photos. Although there is a lot of previous work on this topic, little attention has been paid to zero-shot tasks. Figure 4 As shown in (A) in the figure, when two images do not share semantically similar content, the method of the present invention can generate high-quality translated photos by combining global features and features of dynamic filters, and the details of the content image are well preserved. Figure 4 (B) in Figure 3 shows the translation results when using four seasons images as style images. By examining the four results for each example, we can see that the content is consistent across the four seasons and the appearance style is correct.

[0074] The method of the present invention is applied to video style transfer, such as Figure 5 As shown, Figure 5 This is the result of the different style reference images on consecutive frames of the video by the method of the present invention. The method takes each frame of the video as the content image and trains the network to generate the corresponding style conversion results. Then, the existing deep video prior method is used to improve the temporal correlation of the feature extraction and interpretation of the inventor's model. Figure 5 As shown, the method of the present invention can generate temporally smooth stylized video sequences while well preserving the style features in each frame.

[0075] Figure 6 The figure below is a comparison diagram of the migration results of the method of the present invention with the existing methods when the input style is different painting images. Applying artistic styles to real-life photos is an application widely used by ordinary users in daily life. The method of the present invention can provide users with more freedom because they can use reference images with any artistic style to stylize their photos. Figure 6 As shown, the proposed method produces high-quality results, where the original content structure is maintained and style-related features are appropriately adopted on different objects. Figure 7 In particular, we show style transfer results for sketches. Line drawings are common in artworks, but neural networks have difficulty learning to transfer this style because sketches contain sparse appearance information and significant local structural features, which are averaged out by large amounts of data in traditional transfer methods. Figure 7The first two columns are content images, and the following columns each show the results of existing Avatar Net, AAMS, and our method. Because the other methods did not produce good results, no comparison is made. As can be seen in the zoomed-in window, our method does a great job of preserving the original stroke style.

[0076] Figures 8 to 12 This is an ablation experiment of our method. We study the impact of different components within these modules and evaluate the structural influence of the multi-scale block transformer. We also conduct experiments to analyze how to select the sub-block size for each iteration and the kernel size of the dynamic filtering module.

[0077] Figure 8 For ablation experiments on the network structure, experiments are conducted under different settings, including with and without multi-scale block transformer encoder modules and dynamic filter decoders. Figure 8 (c) in the figure is used as a base model for comparison with the base network (i.e., Cycle GAN) trained in zero-shot learning. Since CycleGAN focuses on learning styles from large unpaired datasets, their zero-shot training has a lot of noise and blur. In order to verify the benefits of the sub-block features extracted by the attention mechanism based on the multi-scale block transformer encoder, Figure 8 (d) and (e) in the figure remove the multi-scale block transformer encoder module, where Figure 8 (d) in the figure is a sub-block provided to the CNN layer to extract features. Figure 8 (e) in the figure is a nested sub-block that is fed into the CNN layer to extract features, which are then combined with global features for decoding. The decoder adaptively interprets shallow features to generate the final result. Figure 8 As shown in (f), the model without dynamic filters produces results with weaker stylization effects. Compared with the results produced by the full model, they are less satisfactory due to the lack of state-of-the-art features. Figure 8 (g) in FIG. 1 is a result obtained by using the method of the present invention. As can be seen from the image, the result of the present method more reasonably transfers the brushstrokes and colors of the target style image.

[0078] Figure 9 The key part is the multi-scale block transformer encoder. If the inventors simply use the transformer encoder to process global features. Figure 9 (a) (b) are the target style image and the original content image. Figure 9 (c) in

[15] only produces weakly stylized results. Figure 9 (d) in the figure evaluates the transformer processing only a single small patch and the internal relations of multiple small patches. Figure 9In (e), by introducing the inter-block transformer, the relationship between sub-blocks of different scales can be learned, thereby improving the stylization effect. Figure 9 (f) in the figure is the complete network, i.e. the migration result obtained using the multi-scale block transformer encoder.

[0079] Figure 10 The selection of the DCM module kernel is verified. It can be seen that when too many or too few kernels are selected, the style is blurred, resulting in the inability to completely transfer the style image.

[0080] Figure 11 and Figure 12 This is an ablation experiment on the size and number of blocks. In the image preprocessing stage, it is necessary to randomly cut a certain number of small blocks from the input image and input them into the encoder. Figure 11 It can be seen that when the number of small blocks is small (less than 4), the generated effect content remains unstable and the style cannot be reasonably transferred. When the number of small blocks selected is greater than or equal to 4, the effect area of ​​style transfer is stable, but the increase in the number of blocks will lead to a rapid increase in graphics card consumption. Therefore, in order to achieve a balance between generation effect and training consumption, 4 nested small blocks are selected in this embodiment of the present invention. Figure 12 Experiments have shown that when the number of blocks is fixed, choosing different block sizes has little effect on the generated results.

[0081] The results of the above comparison and ablation experiments show that the style transfer method proposed in this paper based on generative adversarial networks and transformers for multi-scale block zero-shot learning can reasonably transfer the target style to the original image without the need for large datasets and pre-training models, and has higher practical value than existing methods.

[0082] The above embodiments are only used to illustrate the present invention, and are not intended to limit the present invention. Ordinary technicians in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention. The scope of patent protection of the present invention should be defined by the claims.

Claims

1. A style transfer system based on attention recurrent adversarial network, characterized by: It includes two sets of nested random cropping modules and a cyclic adversarial generation module; the cyclic adversarial generation module includes two sets of generators and their corresponding discriminators; the generator includes a multi-scale block transformer encoder module and a dynamic filtering decoder module; The image nested random cropping module is used to randomly crop each image into multiple nested sub-image blocks; The multi-scale block transformer encoder module includes a block-level transformer encoder and a cross-block-level transformer encoder, which encodes the sub-blocks obtained by the image nested random cropping module to obtain high-dimensional features; The dynamic filtering decoder module is used to deconvolve the high-dimensional features obtained by the multi-scale block transformer encoder module, dynamically filter the shallow high-dimensional features in the deconvolution stage, and adaptively learn the required features to reduce the influence of the original features; The cycle adversarial generation module uses adversarial loss, cycle consistency loss and reconstruction loss to constrain the training of bidirectional mapping to complete the mutual transfer of image styles.

2. A style transfer method based on an attention recurrent adversarial network using the system of claim 1, characterized in that: The following steps are involved: S1: Select two training images A and B of different styles, where A belongs to the x domain and B belongs to the y domain. Use the image nested random cropping module to perform data preprocessing. In each round of training, each image is randomly nested and cropped to obtain K nested sub-block combinations of specified sizes. S2, the preprocessed data obtained in step S1 is reduced in dimension by convolution and the number of channels is increased to obtain block-based convolution features, which are then input into the multi-scale block transformer encoder module for encoding to obtain high-dimensional features and learn the self-attention information within the block and the self-attention information between blocks; S3, using the dynamic filter decoder module to deconvolve the high-dimensional features obtained in step S2, and through deconvolution and upsampling, fuse the global content information of the corresponding dimension layer by layer. In the last layer, a dynamic filter is used to filter the global shallow high-dimensional features, adaptively learn the required shallow content global features, and obtain the image generation result; S4, based on the cycle-consistent adversarial method, simultaneously trains two groups of generators, uses their corresponding discriminator adversarial loss, and then uses the pixel-level L2 paradigm to calculate the cycle-consistent loss and reconstruction loss. The above three losses constrain the training of the generator and discriminator until the network converges, and finally completes the mutual style transfer of training images A and B.

3. The method according to claim 2, characterized in that The step (1) is specifically as follows: in each round of iteration, the training images A and B are randomly nested and cropped to obtain a block combination of K specified side length pixels, and the K cropped sub-blocks are reshaped into an image with a side length of H=W=256 pixels. and 4. The method according to claim 2, characterized in that The step (2) specifically includes the following sub-steps: S201, the pre-processed image obtained in step S1 is reduced in dimension by convolution, the number of channels is increased, and a block feature of size h*w*c is obtained; then it is expanded into a feature sequence of length h*w and dimension c (where h and w are the height and width of the feature map, respectively, and c is the number of channels of the feature map); S202: Send the feature sequence expanded in step S201 to the block-level transformer encoder T p Encode in and learn the sub-block level relationship features; connect the sub-block level relationship features in the channel dimension to generate features of size (h, w, Kc); input the features of size (h, w, Kc) into the cross-block level transformer encoder T c To obtain high-dimensional features, that is, to obtain the relationship features t across multi-scale blocks c .

5. The method according to claim 2, characterized in that The step (3) is specifically as follows: the relationship feature t learned in step S3 c Perform deconvolution, and fuse the global content features of the corresponding dimensions layer by layer through deconvolution and upsampling After upsampling in the third layer, a dynamic filter is embedded to dynamically filter the shallow global content features, and then the filtered global features are fused with the upsampled features; the dynamic filter performs convolution on different convolution kernel sizes according to different inputs. k Calculate the attention weight Π k (x), Input each dynamic convolution submodule for convolution and multiply by the weight π k (x), after mini-batch normalization and relu activation, the adaptive global shallow features are obtained Then perform upsampling again, and finally use multi-layer convolution to reduce the number of channel bits to obtain the image generation results fake_B and fake_A.

6. The method according to claim 5, characterized in that The step (4) specifically includes the following sub-steps: S401, through two groups of discriminators D y , D x Calculate the adversarial loss for the generated results fake_B and fake_A obtained in step S3; the discriminator adversarial loss function is obtained as: Where, x(p k )~P data [x(p)], represents the sampled data from the data distribution P data Sub-block of [x(p)], P data [x(p)] represents the data distribution of image patches with the same style; S402, use cycle-consistent loss to further constrain the training of the two groups of generators G, F and the corresponding discriminator; the generator F generates the image result fake_B obtained by the generator G back to itself, that is, generates rec_A; similarly, the generator G generates the image fake_A generated by the generator F back to itself, that is, generates rec_B; the cycle-consistent loss aims to measure the pixel-level difference loss between rec_A and the training image A through the L1 norm; at the same time, the cycle-consistent loss also includes the L1 loss of rec_B and B; the cycle-consistent loss function is obtained as: In step S403, the two groups of generators introduce reconstruction loss to constrain reverse generation; the reconstruction loss function is obtained as: S404, the objective function is obtained by the above three loss constraints: L(G,F,D x ,D y )=L GAN (G,D y ,x,y)+L GAN (F,D x ,y,x)+λ idt L idt (G,F,x,y)+λ cyc L cyc (G,F,x,y)(4) Among them, λ idt and λ cyc Control the weight between cycle consistency and reconstruction accuracy; After multiple rounds of training, until the generators G and F converge, the target results fake_A and fake_B are finally obtained through iteration.

7. A style transfer device based on attention recurrent adversarial network, characterized in that: The method comprises one or more processors for implementing the style transfer method based on the attention recurrent adversarial network according to any one of claims 1 to 6.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, it is used to implement the style transfer method based on the attention recurrent adversarial network according to any one of claims 1 to 6.

Images