High-quality arbitrary style transfer method for images based on local and global style learning

Abstract

The present invention discloses a method for high-quality arbitrary style transfer of images based on local and global style learning. The method comprises: obtaining a number of content images and images of different style types and constructing them as a training set; establishing a style transfer network including an attention module of local channel perception and a global style fusion module, and constructing a patch-based style transfer loss function; inputting the training set into the style transfer network for training until the patch-based style transfer loss function converges; inputting the content image to be style transferred and the target style type image into the trained style transfer network for processing, outputting a stylized image and displaying it on a display, thereby realizing high-quality style transfer of images. The method of the present invention can better learn from the perspective of local style patterns and global style distribution, and generate high-quality stylized images while performing arbitrary style transfer.

Description

Technical Field

[0001] The present invention relates to a method for high-quality arbitrary style transfer of images, relates to the fields of computer vision and deep learning, and specifically to a method for high-quality arbitrary style transfer of images based on local and global style learning. Background Art

[0002] Style transfer can create new artistic images given arbitrary content and style images. Existing style transfer methods can be mainly divided into: local patch exchange method and global style transfer method. Some existing methods propose to exchange local content patches and their most similar style patches in feature space, such as using attention to integrate style features into content features, and propose to calculate attention-based normalization methods in shallow space and deep space. However, local patch exchange methods may inevitably introduce some semantic information from style images, and the global style distribution of stylized images may deviate from style images. Global style transfer methods mainly adjust the style information of content features from Euclidean distance to match style images, such as adaptive instance normalization methods to adjust the style distribution of content images, and transfer style features to content features through the whitening-coloring process. However, simply using global style transfer methods cannot learn local details of style images. Summary of the invention

[0003] In order to solve the problems existing in the background technology, the present invention provides a method for high-quality arbitrary style transfer of pictures based on local and global style learning. The problem to be solved by the present invention is how to capture local detail textures, avoid introducing unnecessary semantic information from style images, balance local style patterns and global style distribution, and simultaneously consider local style patterns and global style distribution to generate high-quality stylized images.

[0004] The technical solution adopted by the present invention is:

[0005] The method for high-quality arbitrary style transfer of pictures based on local and global style learning of the present invention comprises:

[0006] S1: Obtain several content images and images of different styles and construct them together as a training set.

[0007] S2: Establish a style transfer network based on local and global style learning. The style transfer network includes the local channel-aware attention module LCWA (Local Channel-Wise Attention) and the global style fusion module GSFT (Global Style Fusion Transformer). At the same time, a patch-based style transfer loss function of the style transfer network is constructed. The training set is input into the style transfer network for training until the patch-based style transfer loss function converges to obtain a trained style transfer network.

[0008] S3: The content image to be style-transferred and the target style type image are input into the trained style transfer network for processing. After the processing is completed, the stylized image is output and displayed on the display to achieve high-quality style transfer of the image.

[0009] In step S2, the style transfer network based on local and global style learning includes an encoder of a visual VGG (Visual Geometry Group) model, a local channel-aware attention module LCWA, a global style fusion module GSFT, and a decoder of a visual VGG model connected in sequence. The input of the style transfer network is a content image. I c And a style type picture I s , after being processed by the encoder and the local channel-aware attention module LCWA, the local stylized features are obtained. F cs l , the local stylized features F cs l and style type pictures I s The style of the style type image is transferred to the content image after being processed by the global style fusion module GSFT and the decoder, and the final stylized image is output. I cs .

[0010] The local channel-aware attention module LCWA includes three normalization layers Norm, three convolutional layers conv and a Softmax activation function. I c and style type pictures I s After processing, the content encoding result is obtained F c And style encoding results F s , encode the contentF c Input into the first and third normalization layers Norm for processing respectively, and the style encoding results F s The results of the first and second normalization layers Norm and Norm processing are respectively input into the first and second convolutional layers conv for processing. The results of the first and second convolutional layers conv processing are multiplied and input into the Softmax activation function to output the attention map. A , the first column result X in the feature output after the first convolutional layer conv is processed w 1 β , the second column result X w 2 β and the third column result X w 3 β And the first row of results X in the output features after processing by the second convolutional layer conv w 1 α , the second row result X w 1 α and the third row result X w 1 α Multiplication; the result after the third convolutional layer conv processing V and attention map A After multiplication, we get the first multiplication result and the result after the third convolutional layer conv processing. V The square of V 2 and attention map A After multiplication, the second multiplication result is obtained, which is the average of the second multiplication result and the first multiplication result. M The square of M 2 Subtract and then take the square root to get the standard deviation of the first multiplication result S , the standard deviation of the result after the third normalization layer Norm processing and the first multiplication result S The mean of the dot product and the first multiplication result M Add and finally output local stylized features F cs l As the output of the local channel-aware attention module LCWA.

[0011] The global style fusion module GSFT is a module based on the deep neural network Transformer. The input of the global style fusion module GSFT is a style type image.I s And the local stylized features obtained after processing by the local channel-aware attention module LCWA F cs l , style type pictures I s Input into the deep neural network Transformer for processing, first the style type image I s Divide into several patches and form a style sequence Z s , and then the style sequence is mapped according to the index Q, key K and value V of the predefined deep neural network Transformer Z s Encode, then input into the first multi-head attention module for processing and output the first attention result F s g , the first attention result F s g The fourth normalization layer Norm and the first feed forward neural network FFN are processed in turn, and then encoded by key K and value V, and then input into the second multi-head attention module for processing. The result of the second multi-head attention module and the local stylized feature encoded by index Q are F cs l The residual connection Add and the fifth normalization layer Norm are processed in sequence, and then input into the second and third feedforward neural networks FFN for processing and then output bias respectively. b s and weight w s As the output of the deep neural network Transformer, b s ∈ R b×c×1×1 , w s ∈ R b×c×h×w , b , c , h and w They represent the number of input batches during training, the number of feature channels, the height and width of the features, and the weights. w s and local stylized features F cs lAfter dot multiplication, add the bias b s After adding, the local-global stylized features are output F cs lg As the output of the global style fusion module GSFT.

[0012] The patch-based style transfer loss function includes the content loss and style loss of the visual VGG model, and constructs a patch-based Gram loss (Pctch-wise Gram Loss) algorithm to add to the style loss to establish an improved style loss, using the content loss to transfer the content image I c and stylized images I cs The content similarity is constrained, and the improved style loss is used to classify the style type pictures I s and stylized images I cs The content loss is the content-aware loss function L c , the original style loss includes the second-order style loss function proposed by AdaIN L gs And adversarial loss function L adv .

[0013] The patch-based Gram loss algorithm includes patch operations and Gram matrix loss function L ls , the patching process firstly transforms the style type image I s and stylized images I cs They are randomly cropped, and then image enhancement operations are performed to obtain several style type patches and stylized patches. According to each style type patch and stylized patch, the Gram matrix loss function is input L ls The Gram matrix loss value is obtained for training the style transfer network.

[0014] The Gram matrix loss function L ls as follows:

[0015] L ls =(∑ i N R ( l patch i ,C (I ^i s , I s ), τ )) / N

[0016] R ( l patch i , C (I ^i s , I s ), τ )= l patch i ,when C (I ^i s , I s )≥ τ

[0017] R ( l patch i , C (I ^i s , I s ), τ )=0, others

[0018] C (I ^i s , I s )= G ( E I (I ^i s ))· G ( E I (I s )) / (| G ( E I (I ^i s ))|·| G ( E I (I s ))|)

[0019] in, N Indicates the total number of patches; R ( )and τ Respectively represent the threshold function and its function parameters;l patch i Indicates i The cosine similarity of the Gram matrix of the style type patch and the stylized patch; C ( ) represents the cosine similarity between two tensors; I ^i s Indicates the style type of picture I s No. i Style type patches, ^ As a distinguishing symbol to distinguish the original image and its patch; G ( ) represents the Gram matrix of the third layer in the clipped CLIP space; E I () represents the image encoder for clipping CLIP space.

[0020] The said i Cosine similarity of the Gram matrix of style type patches and stylized patches l patch i as follows:

[0021] l patch i =‖ G ( E I ( aug (I ^i cs )))- G ( E I ( aug (I ^i s )))‖2

[0022] Among them, I ^i cs Represents a stylized image I cs No. i stylized patches; aug ( ) represents random geometric enhancement; ‖ ‖2 represents the second norm.

[0023] The electronic device of the present invention comprises: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method as described above.

[0024] The computer-readable storage medium of the present invention stores program data thereon, and when the program data is executed by a processor, the method described above is implemented.

[0025] The existing attention-based arbitrary style transfer method adopts the attention mechanism to learn style information from the global style signal, ignoring the differences between the content and style signals of different channels and the global. The present invention proposes a style transfer method based on local and global, which can better learn from the perspective of local style patterns and global style distribution. The method uses the local channel-aware attention module LCWA to learn the texture of local details from the style picture, and further learns the global style information through the Transformer-based global style fusion module GSFT to remove some unwanted information in the style picture. Finally, a patch-based style transfer loss function is proposed to balance the local style pattern and the global style distribution.

[0026] The beneficial effects of the present invention are:

[0027] The existing attention-based arbitrary style transfer method adopts the attention mechanism of learning style information in the global style signal, ignoring the differences between the content and style signals of different channels and the global one. The present invention adopts a high-quality arbitrary style transfer method based on local and global style learning, which can better learn from the perspective of local style patterns and global style distribution, and generate high-quality stylized images while performing arbitrary style transfer. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a structural block diagram of the method of the present invention;

[0029] Figure 2 Schematic diagram of the local channel-aware attention module LCWA proposed in the present invention;

[0030] Figure 3 This is a schematic diagram of the Transformer-based global style fusion module GSFT proposed in the present invention;

[0031] Figure 4 Schematic diagram of the patch-based Gram loss algorithm proposed in the present invention;

[0032] Figure 5 : is a comparison diagram of the effects of the embodiment of the present invention and the existing method on image style transfer, wherein: Figure 5 (a) is six pictures of different styles. Figure 5 (b) is six different content pictures. Figure 5 (c) is a stylized image obtained by respectively transferring the styles of six different style type images to six different content images using the method of the present invention. Figure 5(d) shows six stylized images obtained by transferring the styles of different style images to different content images using the diffusion-based image translation using disentangled style and content representation method DiffuselT. Figure 5 (e) is a stylized image obtained by transferring the styles of six different style type images to six different content images using the contrastive coherence preserving loss for versatile style transfer method based on contrast consistency preservation. Figure 5 (f) is a stylized image obtained by transferring the styles of six different style type images to six different content images using the arbitrary style transfer SANet (Arbitrary style transfer with style-attentional networks) method with style attention network. Figure 5 (g) is a stylized image obtained by transferring the styles of six different style type images to six different content images using the transformer-based image style transfer StyTr2 (Image style transfer with transformers) method. Figure 5 (h) is a stylized image obtained by transferring the styles of six different style type images to six different content images using the AdaAttN (Revisit attention mechanism in arbitrary neural style transfer) method in the re-style transfer. Figure 5 (i) is a stylized image obtained by transferring the styles of six different style type images to six different content images using the Artistic style transfer with internal-external learning and contrastive learning (IEST) method. Figure 5(j) is a stylized image obtained by transferring the styles of six different style type images to six different content images using the learning linear transformations for fast image and video style transfer method. Figure 5 (k) is a stylized image obtained by transferring the styles of six different style type images to six different content images using the unbiased image style transfer via reversible neural flows method;

[0033] Figure 6 : is a comparison diagram of the effects of the embodiment of the present invention and the existing method on video style transfer, wherein: Figure 6 (a) is the heat map of the difference between the input content image and style type image and their two adjacent frames. Figure 6 (b) is a heat map of the difference between the stylized image and two adjacent frames obtained by the method of the present invention. Figure 6 (c) is a heat map of the difference between the stylized image and its two adjacent frames obtained by using the image translation DiffuselT method based on the diffusion model and expressing style and content. Figure 6 (d) is a heat map of the difference between the stylized image and its two adjacent frames obtained by using the CCPL method based on comparison consistency preservation. Figure 6 (e) is a heat map of the difference between the stylized image and its two adjacent frames obtained by using the SANet method with a style attention network. Figure 6 (f) is a heat map of the difference between the stylized image and its two adjacent frames obtained by using the transformer-based image style transfer StyTr2 method. Figure 6 (g) is a heat map of the difference between the stylized image and its two adjacent frames obtained by using the attention mechanism AdaAttN method in the re-style transfer. Figure 6 (h) is a heat map of the difference between the stylized image and its two adjacent frames obtained by using the artistic style transfer IEST method based on internal and external learning and comparison learning. Figure 6 (i) is a heat map of the difference between the stylized image and its two adjacent frames obtained by using the linear transformation for fast image and video style transfer Linear method. Figure 6(j) is a heat map of the difference between the stylized image and its two adjacent frames obtained by unbiased image style transfer through the reversible neural flow ArtFlow method. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present invention will be described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The method for high-quality arbitrary style transfer of pictures based on local and global style learning of the present invention is specifically as follows:

[0036] like Figure 1 As shown in the figure, a style transfer network based on local and global style learning is established. The style transfer network based on local and global style learning includes an encoder of a visual VGG model, a local channel-aware attention module LCWA, a global style fusion module GSFT and a decoder of a visual VGG model connected in sequence. The input of the style transfer network is a content image. I c And a style type picture I s , after being processed by the encoder and the local channel-aware attention module LCWA, the local stylized features are obtained. F cs l , local stylized features F cs l and style type pictures I s The style of the style type image is transferred to the content image after being processed by the global style fusion module GSFT and the decoder, and the final stylized image is output. I cs .

[0037] like Figure 2 As shown in Figure 2, the local channel-aware attention module LCWA includes three normalization layers Norm, three 1×1 convolutional layers conv and a Softmax activation function. I c and style type pictures I s After processing, the content encoding result is obtained F c And style encoding results F s, the content encoding result F c Input into the first and third normalization layers Norm for processing respectively, and the style encoding results F s The results are respectively input into the second normalization layer Norm and the third 1×1 convolution layer conv for processing. The results of the first and second normalization layers Norm are respectively input into the first and second 1×1 convolution layers conv for processing. The results of the first and second 1×1 convolution layers conv are multiplied and input into the Softmax activation function to output the attention map. A , the first column of the output feature after the first 1×1 convolutional layer conv is processed w 1 β , the second column result X w 2 β and the third column result X w 3 β And the first row of the output feature after the second 1×1 convolutional layer conv is processed w 1 α , the second row result X w 1 α and the third row result X w 1 α Multiplication; the result after the third 1×1 convolutional layer conv V and attention map A After multiplication, we get the first multiplication result and the result after the third 1×1 convolutional layer conv processing. V The square of V 2 and attention map A After multiplication, the second multiplication result is obtained, which is the average of the second multiplication result and the first multiplication result. M The square of M 2 Subtract and then take the square root to get the standard deviation of the first multiplication result S , the standard deviation of the result after the third normalization layer Norm processing and the first multiplication result S The mean of the dot product and the first multiplication result M Add and finally output local stylized features F cs l As the output of the local channel-aware attention module LCWA.

[0038] The visual VGG-19 model is often used in any style transfer to extract salient features for identification. While it is effective to use the visual VGG-19 model to extract style features and fuse them into content features, it ignores the diversity of different local patterns between different channels. When obtaining the attention map, the visual VGG-19 model does not consider the content features extracted by the visual VGG-19 model. Q and style features K The importance of different channels will simply assign weights to each channel uniformly; the present invention uses content and style to learn parameters W con and W sty Expand and redistribute the weights for each channel of the feature. W con 、W sty ∈ R B×512×512 , B Indicates the number of inputs sent to the network at one time. In specific implementation, B=8. It can dynamically assign different weights to different channels of style features, and pay more attention to local style features from the perspective of channels. The local channel-aware attention module LCWA can perceive local style patterns and integrate them into content features. It learns local style patterns between different style feature channels, pays attention to different channels, and avoids ignoring important local style patterns.

[0039] like Figure 3 As shown in Figure 1, the global style fusion module GSFT is a module based on the deep neural network Transformer. The input of the global style fusion module GSFT is a style type image. I s And the local stylized features obtained after processing by the local channel-aware attention module LCWA F cs l , style type pictures I s Input into the deep neural network Transformer for processing, first the style type image I s Divide into several patches and form a style sequence Z s , and then the style sequence is mapped according to the index Q, key K and value V of the predefined deep neural network Transformer Z s Encode, then input into the first multi-head attention module for processing and output the first attention result F s g, the first attention result F s g The fourth normalization layer Norm and the first feed-forward neural network FFN are input in turn for processing, and then encoded by key K and value V, and then input into the second multi-head attention module for processing. The result after the second multi-head attention module and the local stylized feature after index Q encoding are F cs l The residual connection Add and the fifth normalization layer Norm are processed in sequence, and then input into the second and third feedforward neural networks FFN for processing and then output bias respectively. b s and weight w s As the output of the deep neural network Transformer, b s ∈ R b×c×1×1 , w s ∈ R b×c×h×w , b , c , h and w They represent the number of input batches during training, the number of feature channels, and the height and width weights of the features. w s and local stylized features F cs l After dot multiplication, add the bias b s After adding, the local-global stylized features are output F cs lg As the output of the global style fusion module GSFT.

[0040] The attention module LCWA using local channel perception can only learn local style features, but cannot learn global style features. Therefore, the present invention uses a deep neural network Transformer to learn global style information, mainly by transforming style type images into I s Divide into a style sequence Z s , and then use the multi-head attention mechanism to learn the long-range dependency of style information and generate weights w s and bias b s , using weights w s and biasb s Reprocess the learned local stylized features F cs l , and finally obtain local-global stylized features F cs lg as input to the decoder.

[0041] The Transformer-based global style fusion module GSFT focuses more on exploring how to integrate global style features with local stylized features. It can use Transformer to reintegrate the learned global style features into local stylized features, further maintaining the style consistency between stylized images and style images.

[0042] like Figure 1 and Figure 4 As shown in the figure, a patch-based style transfer loss function of the style transfer network is constructed, including the content loss and style loss of the visual VGG model, and a patch-based Gram loss algorithm is constructed to add to the style loss to establish an improved style loss. The content loss is used to improve the content image. I c and stylized images I cs The content similarity is constrained, and the improved style loss is used to classify the style type pictures I s and stylized images I cs The content loss is the content-aware loss function L c , the original style loss includes the second-order style loss function proposed by AdaIN L gs And adversarial loss function L adv .

[0043] The patch-based Gram loss algorithm includes the patch operation and the Gram matrix loss function. L ls , the patching process firstly transforms the style type image I s and stylized images I cs They are randomly cropped, and then image enhancement operations are performed to obtain several style type patches and stylized patches. According to each style type patch and stylized patch, the Gram matrix loss function is input L ls The Gram matrix loss function is used to obtain the Gram matrix loss value for training the style transfer network.L ls Below:

[0044] L ls =(∑ i N R ( l patch i , C (I ^i s , I s ), τ )) / N

[0045] R ( l patch i , C (I ^i s , I s ), τ )= l patch i ,This C (I ^i s , I s ) ≥ τ

[0046] R ( l patch i , C (I ^i s , I s ), τ )=0, its

[0047] C (I ^i s , I s )= G ( E I (I ^i s ))· G ( E I (I s )) / (| G ( E I (I ^i s))|·| G ( E I (I s ))|)

[0048] l patch i =‖ G ( E I ( aug (I ^i cs )))- G ( E I ( aug (I ^i s )))‖2

[0049] in, N Indicates the total number of patches; R ( )and τ Respectively represent the threshold function and its function parameters. τ =0.75; l patch i Indicates i The cosine similarity of the Gram matrix of the style type patch and the stylized patch; C ( ) represents the cosine similarity between two tensors; I ^i s Indicates the style type of picture I s No. i Style type patches, ^ As a distinguishing symbol to distinguish the original image and its patch; G ( ) represents the Gram matrix of the third layer in the clipped CLIP space; E I () represents the image encoder for clipping CLIP space; I ^i cs Represents a stylized image I cs No. i stylized patches; aug ( ) represents random geometric enhancement; ‖ ‖2 represents the second norm.

[0050] The style loss function currently used constrains the similarity between content and style from a global perspective, while ignoring local style information. The patch-based Gram loss algorithm of the present invention improves the quality of stylized images by regularizing local style patterns between stylized patterns, thereby improving the quality of stylized images from style type images.I s and stylized images I cs Randomly extract the same number of style patches from the stylized image I cs Patch and style type images extracted from I s The patches are in the clipped CLIP space and the Gram matrix is ​​calculated, and their Euclidean distances are constrained.

[0051] In order to learn the local style information of the style image, the present invention proposes a patch-based Gram loss algorithm. Specifically, it is to learn the local style information of the style image by I s and stylized images I cs The images are randomly cropped and the cropped images are subjected to image enhancement operations such as flipping in the same way. However, due to the different style types of images obtained by each cropping, I s and stylized images I cs It cannot fully represent the global style of the original image, so some patches with low similarity to the original image are removed. Specifically, the cosine similarity between the Gram matrix of each style patch and the style image is calculated. l patch i .

[0052] The content dataset MS-COCO and the style type dataset WikiArt are jointly constructed as training sets, and the training sets are input into the style transfer network for training. During the training process, all images are resized to 512×512, and then randomly cropped to a fixed resolution of 256×256. At the same time, the Adam optimizer learning rate is set to 0.0005, and the batch size and iteration number are set to 8 and 160000 respectively. All training is built on the NVIDIA RTX 3090 (24GB) GPU using the Pytorch framework. Training is performed until the patch-based style transfer loss function converges to the minimum, and the trained style transfer network is obtained.

[0053] Finally, the content image to be style-transferred and the target style type image are input into the trained style transfer network for processing. After the processing, the stylized image is output and displayed on the monitor to achieve high-quality style transfer of the image. Figure 5 (a) Figure 5 (b) Figure 5 (c) Figure 5 (d) Figure 5 (e), Figure 5(f), Figure 5 (g), Figure 5 of (h), Figure 5 (i) Figure 5 (j) and Figure 5 As shown in (k), there are six different style type pictures, six different content pictures, and the method of the present invention, the image translation DiffuselT method based on the diffusion model and expressing style and content, the universal style transfer CCPL method based on comparison consistency preservation, the arbitrary style transfer SANet method with style attention network, the transformer-based picture style transfer StyTr2 method, the attention mechanism AdaAttN method in the re-style transfer, the art style transfer IEST method based on internal and external learning and comparison learning, the Linear method for learning linear transformation for fast image and video style transfer, and the unbiased image style transfer method. The styles of six different style type pictures are respectively transferred to six different content pictures through the reversible neural flow ArtFlow method. The style of the present invention is compared with the effects of each method on picture style transfer during specific implementation. As shown Figure 6 (a) Figure 6 (b) Figure 6 (c) Figure 6 (d) Figure 6 (e), Figure 6 (f), Figure 6 (g), Figure 6 of (h), Figure 6 (i) and Figure 6 As shown in (j), in a specific implementation, the present invention extracts video frames from a video and compares the effects of various methods on video style migration, wherein: Figure 6 The first picture in (a) is the input picture, the large picture is the content picture, and the small picture is the style type picture. According to the heat map of the difference between two adjacent frames of each picture, the similarity LPIPS (Learned Perceptual Image Patch Similarity) index is calculated to evaluate the perceptual similarity between the generated image and the real image. The similarity LPIPS indexes of the stylized picture and the content image obtained by the method of the present invention and other methods are 0.3105, 0.4206, 0.3679, 0.3273, 0.3792, 0.3253, 0.3485, 0.3371 and 0.3343 respectively. The similarity LPIPS index of the stylized picture and the content image obtained by the method of the present invention is the lowest at 0.3105, which has the best effect. It can be seen that the method of the present invention can better learn from the perspective of local style mode and global style distribution, and generate high-quality stylized pictures while performing arbitrary style transfer.

[0054] Most of the existing attention-based methods process the features extracted by the VGG model. The local channel-aware attention module LCWA proposed in the present invention fuses local style patterns into content features to obtain local stylized features. However, relying solely on the local channel-aware attention module LCWA to focus on limited areas of local patterns in the style image may lead to deviations from the global style distribution. To this end, the present invention proposes a Transformer-based global style fusion module GSFT to solve this problem. A Transformer is used to learn the global style distribution and dynamically generate global style weights and biases to pull out local stylized features aligned with it. Considering that the currently widely used style loss is based on second-order global style statistics, this may cause the framework proposed in the present invention to tend to learn more global style distributions. To this end, the present invention proposes a patch-based style transfer loss function to improve the quality of stylized images by regularizing local style patterns between stylized patterns.

[0055] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0056] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A high-quality arbitrary style transfer method for images based on local and global style learning, characterized in that: include: S1: Obtain several content images and images of different styles and construct them together as a training set; S2: Establish a style transfer network based on local and global style learning. The style transfer network includes the local channel-aware attention module LCWA and the global style fusion module GSFT. At the same time, a patch-based style transfer loss function of the style transfer network is constructed. The training set is input into the style transfer network for training until the patch-based style transfer loss function converges, and a trained style transfer network is obtained. S3: The content image to be style-transferred and the target style type image are input into the trained style transfer network for processing. After the processing is completed, the stylized image is output and displayed on the display, so as to achieve high-quality style transfer of the image; In step S2, the style transfer network based on local and global style learning includes an encoder of a visual VGG model, a local channel-aware attention module LCWA, a global style fusion module GSFT and a decoder of a visual VGG model connected in sequence, and the input of the style transfer network is a content image. I c And a style type picture I s , after being processed by the encoder and the local channel-aware attention module LCWA, the local stylized features are obtained. F cs l , the local stylized features F cs l and style type pictures I s The style of the style type image is transferred to the content image after being processed by the global style fusion module GSFT and the decoder, and the final stylized image is output. I cs ; The local channel-aware attention module LCWA includes three normalization layers Norm, three convolutional layers conv and a Softmax activation function. I c and style type pictures I s After processing, the content encoding result is obtained F c And style encoding results F s , the content encoding result F c Input into the first and third normalization layers Norm for processing respectively, and the style encoding results F s The results of the first and second normalization layers Norm and Norm processing are respectively input into the first and second convolutional layers conv for processing. The results of the first and second convolutional layers conv processing are multiplied and input into the Softmax activation function to output the attention map. A , the result after the third convolutional layer conv processing V and attention map A After multiplication, we get the first multiplication result and the result after the third convolutional layer conv processing. V The square of V 2 and attention map A After multiplication, the second multiplication result is obtained, which is the average of the second multiplication result and the first multiplication result. M The square of M 2 Subtract and then take the square root to get the standard deviation of the first multiplication result S , the standard deviation of the result after the third normalization layer Norm processing and the first multiplication result S The mean of the dot product and the first multiplication result M Add and finally output local stylized features F cs l As the output of the local channel-aware attention module LCWA.

2. The method for high-quality arbitrary style transfer of images based on local and global style learning according to claim 1, characterized in that: The global style fusion module GSFT is a module based on the deep neural network Transformer. The input of the global style fusion module GSFT is a style type image. I s And the local stylized features obtained after processing by the local channel-aware attention module LCWA F cs l , style type pictures I s Input into the deep neural network Transformer for processing, first the style type image I s Divide into several patches and form a style sequence Z s , and then the style sequence is mapped according to the index Q, key K and value V of the predefined deep neural network Transformer Z s Encode, then input into the first multi-head attention module for processing and output the first attention result F s g , the first attention result F s g The fourth normalization layer Norm and the first feed-forward neural network FFN are input in turn for processing, and then encoded by key K and value V, and then input into the second multi-head attention module for processing. The result after the second multi-head attention module and the local stylized feature after index Q encoding are F cs l The residual connection Add and the fifth normalization layer Norm are processed in sequence, and then input into the second and third feedforward neural networks FFN for processing and then output bias respectively. b s and weight w s As the output of the deep neural network Transformer, the weight w s and local stylized features F cs l After dot multiplication, add the bias b s After adding, the local-global stylized features are output F cs lg As the output of the global style fusion module GSFT.

3. The method for high-quality arbitrary style transfer of images based on local and global style learning according to claim 1, characterized in that: The patch-based style transfer loss function includes the content loss and style loss of the visual VGG model, and constructs a patch-based Gram loss algorithm to add the style loss to establish an improved style loss, using the content loss to transfer the content image I c and stylized images I cs The content similarity is constrained, and the improved style loss is used to classify the style type pictures I s and stylized images I cs The style similarity is constrained.

4. The method for high-quality arbitrary style transfer of images based on local and global style learning according to claim 3, characterized in that: The patch-based Gram loss algorithm includes patch operations and Gram matrix loss function L ls , the patching process firstly transforms the style type image I s and stylized images I cs They are randomly cropped, and then image enhancement operations are performed to obtain several style type patches and stylized patches. According to each style type patch and stylized patch, the Gram matrix loss function is input L ls The Gram matrix loss value is obtained for training the style transfer network.

5. The method for high-quality arbitrary style transfer of images based on local and global style learning according to claim 4, characterized in that: The Gram matrix loss function L ls as follows: L ls =(∑ i N   R ( l patch i , C (I ^i s , I s ), τ )) / N R ( l patch i , C (I ^i s , I s ), τ )= l patch i ,This C (I ^i s , I s ) ≥ τ R ( l patch i , C (I ^i s , I s ), τ )=0, others C (I ^i s , I s )= G ( E I (I ^i s ))· G ( E I (I s )) / (| G ( E I (I ^i s ))|·| G ( E I (I s ))|) in, N Indicates the total number of patches; R ( )and τ Respectively represent the threshold function and its function parameters; l patch i Indicates i The cosine similarity of the Gram matrix of the style type patch and the stylized patch; C ( ) represents the cosine similarity between two tensors; I ^i s Indicates the style type of picture I s No. i Style type patches; G ( ) represents the Gram matrix of the third layer in the clipped CLIP space; E I () represents the image encoder for clipping CLIP space.

6. The method for high-quality arbitrary style transfer of images based on local and global style learning according to claim 5, characterized in that: The said i Cosine similarity of the Gram matrix of style type patches and stylized patches l patch i as follows: l patch i =‖ G ( E I ( aug (I ^i cs )))- G ( E I ( aug (I ^i s )))‖2 Among them, I ^i cs Represents a stylized image I cs No. i stylized patches; aug ( ) represents random geometric enhancement; ‖‖2 represents the second norm.

7. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images