Asymmetric image style migration method based on Mamba-GAN network and Zigzag scanning

Abstract

The invention relates to an asymmetric image style migration method based on a Mama-GAN network and Zigzag scanning, and the method comprises the steps: fusing the Mama network and a GAN structure, employing the asymmetric training capability of the GAN, reducing the dependence on GPU resources, and avoiding the demands of pairwise training data; the generator adopts a Zigzag Patch Embedding strategy and a multi-scale feature fusion module, and is combined with a Zigzag scanning mechanism, so that image details are effectively reserved, and a style migration effect is enhanced; a multi-discriminator structure is adopted, so that the authenticity of the generated image is further improved; the method is excellent in performance on a plurality of artistic style data sets, not only realizes high-quality style migration, but also is obviously superior to an existing algorithm, and particularly provides a solution which is efficient, low in resource consumption and high in detail processing capability for the field of image style migration in the aspects of detail reservation and resource consumption.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an asymmetric image style transfer method based on a Mamba-GAN network and Zigzag scanning. Background Art

[0002] Image style transfer is one of the research directions that has attracted much attention in the field of computer vision and image processing in recent years. It mainly uses deep learning, especially neural network models, to apply the style of one image to another by separating and recombining the content image and the style image. With the development of deep learning technology, a variety of different image style transfer methods have emerged. In asymmetric image style transfer, the following methods are usually used: (1) The traditional Cycle-GAN method integrates ordinary convolutional neural networks (CNNs). CNNs have achieved great success in tasks such as image recognition and classification, but their feature extraction capabilities are limited in image style transfer. This causes the Cycle-GAN method to easily lose some detail information when dealing with asymmetric image style transfer tasks and fail to accurately capture local texture features, resulting in the details of the produced image not being realistic enough; (2) Transformer-based image style transfer Style transfer methods, such as the Str2 method, are due to the powerful feature processing capabilities of the Transformer network structure, which can capture long-distance dependencies in images. However, training the Transformer requires a large amount of computing resources, especially a high degree of dependence on GPU resources. This makes the Transformer-based methods face problems such as high hardware costs and long training time in practical applications. At the same time, such methods will still lose some detail information when processing complex images; (3) Image style transfer methods based on the Diffusion model. The Diffusion model generates high-quality images by gradually adding noise and denoising processes, which can better compensate for the problem of missing details and generate more realistic images. However, the method based on the Diffusion model also has a huge demand for GPU resources, which limits its wide application. Summary of the Invention

[0003] In view of this, the present invention provides an asymmetric image style transfer method based on the Mamba-GAN network and Zigzag scanning, which can reduce the dependence of asymmetric image style transfer on GPU resources and enhance the detail preservation capability in style transfer.

[0004] To achieve the above objectives, the present invention provides an asymmetric image style transfer method based on a Mamba-GAN network and Zigzag scanning, comprising the following steps:

[0005] S1. Build the Mamba-GAN network, including the input layer, Zigma Patch Embedding layer, linear projection module, generator, discriminator, and output layer;

[0006] S2, input the unpaired image into the Mamba-GAN network,

[0007] S201, using the Zigma Patch Embedding layer to scan the input unpaired image using Zigzag scanning, dividing the unpaired image into multiple patches and converting them into feature vectors, and recording the location information of the patches;

[0008] S202: Input the feature vectors of the multiple patches into a linear projection module, extract image features through linear transformation, and input them into a generator of the Mamba-GAN network;

[0009] S203, the generator of the Mamba-GAN network includes a Mamba Block layer and a decoder, and the input features are nonlinearly transformed through the Mamba Block layer to capture high-level features of the image;

[0010] The decoder includes a reverse Patch Embedding layer and multiple convolutional layers. The reverse Patch Embedding layer restores the image data that has undergone linear projection in the Zigma Patch Embedding layer and recovers the original patch encoding position information of the image. The multiple convolutional layers then adjust the size and resolution of the feature map and upsample to generate an image with the same resolution as the original image.

[0011] S204, inputting the generated image into the discriminator for discrimination;

[0012] S205: Output the final generated image through the output layer.

[0013] Preferably, the discriminator includes three standard discriminators, which are stacked to form the discriminator.

[0014] Preferably, the Mamba Block layer includes two paths, a gating branch and a state space model SSM branch. The state space model SSM branch includes two branches. One branch fuses the input image feature X with the output image feature of the gated branch activation function layer through a forward propagation convolution layer and a forward SSM layer to obtain an image feature X1. The other branch fuses the image feature X1 with the image feature X2 through a forward propagation convolution layer and a backward SSM layer. The image feature X2 is fused with the input image feature to generate the output of the Mamba Block layer.

[0015] Preferably, the loss function of the Mamba-GAN network includes content loss and discrimination loss; the content loss is the difference between the features extracted from the input unmatched image and the image output by the Mamba-GAN network generator after the pre-trained VGG network, and is expressed as:

[0016] L c =∑MSE(V i (G(Z(X))),V i (X)),i=1,2,3,…,n

[0017] Among them, L c represents the content loss, X represents the input unmatched image, V i represents the i-th VGG network subnetwork, Z represents Zigzag scanning, G(.) represents the output image of the Mamba-GAN network generator, and MSE represents the squared error loss;

[0018] The discriminant loss expression is:

[0019] L D =-E x～pdt [logD(x)]-E z～pz [log(1-D(G(z)))]

[0020] Among them, E x～pdt represents the sampling real data distribution, E z～pz represents the sampled self-noise distribution, D(.) represents the output image of the Mamba-GAN network generator;

[0021] The loss function expression of the Mamba-GAN network is:

[0022] L G =γ1L c +γ2L D

[0023] Among them, γ1 and γ2 represent weights.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] This paper achieves efficient asymmetric image style transfer by integrating the Mamba network and the GAN network structure and introducing a Zigzag scanning mechanism. The Mamba-GAN network architecture utilizes its unique Mamba Block and multi-discriminator structure to significantly improve the image style transfer effect and detail processing capabilities while reducing dependence on GPU resources. The Zigzag scanning mechanism further optimizes the spatial continuity of the image, allowing the generated image to better preserve detailed features during the style transfer process.

[0026] The technical solution provided by the present invention can effectively realize the style transfer between Horse2Zebra, Orange2Apple and Summer2Winter, showing good style conversion effect, which is better than the existing Cycle-GAN algorithm and also performs well in detail preservation. At the same time, the required training resources are far lower than those of the Transformer-based model. In addition, through experiments comparing different Zigzag scanning strategies, the positive effect of the left-to-right Zigzag scanning method on improving network performance is verified. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of the Mamba-GAN network structure of the present invention;

[0028] Figure 2 This is a schematic diagram of the Mamba Block network structure of the present invention;

[0029] Figure 3 Schematic diagram of the decoder structure of the present invention;

[0030] Figure 4 Schematic diagram of the ZigZag scanning strategy of the present invention;

[0031] Figure 5 This is the overall effect diagram of the Unpaired Mamba-GAN image style transfer of the present invention;

[0032] Figure 6 This is a comparison chart of the Vangoh image style transfer effect of the present invention;

[0033] Figure 7 This is a comparison chart of the Cézanne image style transfer effect of the present invention;

[0034] Figure 8 This is a comparison chart of the effects of monent image style migration in the present invention;

[0035] Figure 9 This is a comparison chart of the Ukiyo-e image style transfer effect of the present invention;

[0036] Figure 10 This is a comparison diagram of the Zigzag scanning method of the present invention. DETAILED DESCRIPTION

[0037] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0038] This embodiment provides an asymmetric image style transfer method based on a Mamba-GAN network and Zigzag scanning, including the following steps:

[0039] S1. Build the Mamba-GAN network, such as Figure 1 As shown, it includes the input layer, Zigma Patch Embedding layer, linear projection module, generator, discriminator and output layer;

[0040] S2, input the unpaired image into the Mamba-GAN network,

[0041] S201, using the Zigma Patch Embedding layer to scan the input unpaired image using Zigzag scanning, dividing the unpaired image into multiple patches and converting them into feature vectors, and recording the location information of the patches;

[0042] The traditional Patch Embedding mechanism usually slices the image repeatedly from left to right, which easily destroys the spatial continuity of the input image. The Zigma algorithm introduces a zigzag scanning method, such as Figure 4 As shown in FIG, a traditional scanning method and two zigzag scanning strategies are shown: a zigzag scan from left to right and a zigzag scan from top to bottom. This embodiment draws on the idea of ​​zigzag scanning to improve the effect of image style transfer.

[0043] S202: Input the feature vectors of the multiple patches into a linear projection module, extract image features through linear transformation, and input them into a generator of the Mamba-GAN network;

[0044] S203, the generator of the Mamba-GAN network includes a Mamba Block layer and a decoder, and the input features are nonlinearly transformed through the Mamba Block layer to capture high-level features of the image;

[0045] like Figure 2 As shown, Mamba Block is composed of the standard Mamba module in Vision Mamba. The MambaBlock layer includes two paths: a gating branch and a state-space model SSM branch. The state-space model SSM branch includes two branches. One branch fuses the input image feature X with the output image feature of the gated branch activation function layer through a forward propagation convolution layer and a forward SSM layer to obtain image feature X1. The other branch fuses the image feature X1 with the image feature X1 through a forward propagation convolution layer and a backward SSM layer to obtain image feature X2. The image feature X2 is fused with the input image feature to generate the output of the Mamba Block layer.

[0046] like Figure 3 As shown, the decoder includes a reverse patch embedding layer and multiple convolutional layers. The reverse patch embedding layer restores the image data that has undergone linear projection in the zigma patch embedding layer and recovers the original patch encoding position information of the image. The multiple convolutional layers then adjust the size and resolution of the feature map and upsample to generate an image with the same resolution as the original image.

[0047] S204: Input the generated image into the discriminator for discrimination. The three standard discriminators are stacked to form the discriminator.

[0048] The loss function of the Mamba-GAN network includes content loss and discrimination loss. The content loss is the difference between the features extracted from the input unmatched image and the image output by the Mamba-GAN network generator after the pre-trained VGG network. The expression is:

[0049] L c =∑MSE(V i (G(Z(X))),V i (X)),i=1,2,3,…,n

[0050] Among them, L c represents the content loss, X represents the input unmatched image, V i represents the i-th VGG network subnetwork, Z represents Zigzag scanning, G(.) represents the output image of the Mamba-GAN network generator, and MSE represents the squared error loss;

[0051] The discriminant loss expression is:

[0052] L D =-E x～pdt [logD(x)]-E z～pz [log(1-D(G(z)))]

[0053] Among them, E x～pdt represents the sampling real data distribution, E z～pz represents the sampled self-noise distribution, D(.) represents the output image of the Mamba-GAN network generator;

[0054] The loss function expression of the Mamba-GAN network is:

[0055] L G =γ1L c +γ2L D

[0056] Among them, γ1 and γ2 represent weights

[0057] S205: Output the final generated image through the output layer.

[0058] The performance of the image style transfer method proposed in this example is evaluated using a dataset from Cycle-GAN. The dataset includes several subsets featuring four typical artistic styles: Monet, Van Gogh, Cézanne, and Ukiyo-e, as well as corresponding natural photographs. Each subset contains a large number of images from both the source and target domains. In addition, to evaluate the generalization ability of the proposed algorithm under different semantic visual structures, three other classic image-to-image translation datasets proposed by Cycle-GAN are also used: Horse2Zebra, Orange2Apple, and Summer2Winter. These datasets are all unpaired.

[0059] This example is trained on an NVIDIA 4090D GPU with 24GB of memory. The deep training environment uses Python 3.9 and PyTorch 2.0.0+cu118. When training the generator in Mamba-GAN, the number of stacked discriminators m and the generator and discriminator weights γ1 and γ2 are set as shown in Table 1:

[0060] Table 1. Parameter value settings in Mamba-GAN

[0061] Parameter name describe Experimental values n Number of stacked discriminators 3 <![CDATA[γ1]]> Content loss 1 <![CDATA[γ2]]> Discriminative loss 5

[0062] The relevant parameter settings for training the Mamba-GAN network are shown in Table 2:

[0063] Table 2. Parameter settings for training the Mamba-GAN network

[0064] Parameter name describe Experimental values batch_size Training batch size 4 epoch Number of training iterations 160000 lr Initial value of learning rate 0.0001

[0065] (1) Comparison with Unpaired Mamba-GAN

[0066] The overall results of the unpaired Mamba-GAN are as follows Figure 5 As shown in Figure 2, good style transfer is effectively achieved between the Horse2Zebra, Summer2Winter, and Orange2Apple datasets; comparisons are made on four different art style datasets: Monet, Van Gogh, Cézanne, and Ukiyo-e; the proposed method is compared with the unpaired Cycle-GAN algorithm and the most advanced Transformer-based algorithm StyTr2 (which is based on paired training). The experimental results are shown in Figure 2. Figures 6 to 9As shown in Figure 3, in all four datasets, the proposed algorithm not only achieves the best style transfer effect, but also effectively preserves fine-grained detail features;

[0067] (2) Comparison of zigzag scanning mechanisms

[0068] In order to evaluate the impact of different zigzag scanning strategies on the performance of the Mamba-GAN network in image style transfer, this implementation adopts three different scanning methods, namely the traditional zigzag-free method, the top-to-bottom zigzag scanning method, and the left-to-right zigzag scanning method; the experiments on the Van Gogh dataset are as follows: Figure 10 As shown in the figure: the zigzag scanning method from top to bottom has the worst effect and cannot preserve the characteristics of the wooden bridge in the image; the zigzag scanning from left to right and the traditional method both successfully preserve the characteristics of the wooden bridge; in terms of overall style transfer performance, the image generated by the zigzag scanning method from left to right is more representative of Van Gogh's style; this shows that the zigzag scanning strategy from left to right has a certain positive impact on the performance of the Mamba-GAN network.

[0069] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as above in terms of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can, without departing from the scope of the technical solution of the present invention, make some changes or modifications to equivalent embodiments using the technical contents disclosed above. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. Asymmetric image style transfer method based on Mamba-GAN network and Zigzag scanning, characterized by: The following steps are involved: S1. Build the Mamba-GAN network, including the input layer, Zigma Patch Embedding layer, linear projection module, generator, discriminator, and output layer; S2, input the unpaired image into the Mamba-GAN network, S201, using the Zigma Patch Embedding layer to scan the input unpaired image using Zigzag scanning, dividing the unpaired image into multiple patches and converting them into feature vectors, and recording the location information of the patches; S202: Input the feature vectors of the multiple patches into a linear projection module, extract image features through linear transformation, and input them into a generator of the Mamba-GAN network; S203, the generator of the Mamba-GAN network includes a Mamba Block layer and a decoder, and the input features are nonlinearly transformed through the MambaBlock layer to capture high-level features of the image; The decoder includes a reverse Patch Embedding layer and multiple convolutional layers. The reverse Patch Embedding layer restores the image data that has undergone linear projection in the Zigma Patch Embedding layer and recovers the original patch encoding position information of the image. The multiple convolutional layers then adjust the size and resolution of the feature map and upsample to generate an image with the same resolution as the original image. S204, inputting the generated image into the discriminator for discrimination; S205: Output the final generated image through the output layer.

2. The asymmetric image style transfer method based on Mamba-GAN network and Zigzag scanning according to claim 1, characterized in that: The discriminator includes three standard discriminators, which are stacked to form a discriminator.

3. The asymmetric image style transfer method based on Mamba-GAN network and Zigzag scanning according to claim 1, characterized in that: The Mamba Block layer includes two paths: a gating branch and a state-space model SSM branch. The state-space model SSM branch includes two branches. One branch fuses the input image feature X with the output image feature of the gated branch activation function layer through a forward propagation convolution layer and a forward SSM layer to obtain an image feature X1. The other branch fuses the image feature X1 with the forward propagation convolution layer and the backward SSM layer to obtain an image feature X2. The image feature X2 is fused with the input image feature to generate the output of the Mamba Block layer.

4. The asymmetric image style transfer method based on Mamba-GAN network and Zigzag scanning according to claim 1, characterized in that: The loss function of the Mamba-GAN network includes content loss and discrimination loss. The content loss is the difference between the features extracted from the input unmatched image and the image output by the Mamba-GAN network generator after the pre-trained VGG network, and is expressed as: L c =∑MSE(V i (G(Z(X))),V i (X)),i=1,2,3,…,n Among them, L c represents the content loss, X represents the input unmatched image, V i represents the i-th VGG network subnetwork, Z represents Zigzag scanning, G(.) represents the output image of the Mamba-GAN network generator, and MSE represents the squared error loss; The discriminant loss expression is: L D =-E x～pdt [logD(x)]-E z～pz [log(1-D(G(z)))] Among them, E x～pdt represents the sampling real data distribution, E z～pz represents the sampled self-noise distribution, D(.) represents the output image of the Mamba-GAN network generator; The loss function expression of the Mamba-GAN network is: L G =γ1L c +γ2L D Among them, γ1 and γ2 represent weights.

Images